By Caitlin Palmer, Senior Strategy & Operations Manager
Today DoorDash is expanding delivery to 47 new neighborhoods in the Tri-State area. Our expanded delivery area in the NY metro region includes Staten Island; Hoboken, New Brunswick and North Bergen in New Jersey; Astoria, Long Island City and Woodside in Queens; Sunset Park, Bay Ridge and Sheepshead Bay in South Brooklyn; Huntington, Smithtown, Babylon and Islip in Long Island Suffolk County; Central and South Westchester; andStamford, Greenwich, Norwalk, Westport, Fairfield and Bridgeport, Connecticut. You can see a full list here.
To celebrate, DoorDash is offering free delivery for all new customers on their first order (automatically applied at checkout), as well as $5 off orders of $20 or more with the code LAUNCH5 in New Jersey, New York and Connecticut.
We’re thrilled to bring the convenience of delivery to so many new areas in the Tri-State area, so if you’re hungry get started today at www.doordash.com. If you have friends or co-workers in our newly launched areas, refer them to try DoorDash — they get $7 referral credit and so do you!
From DoorDash’s very start, we’ve seen Dashers as one of our key customers. We aim to provide the best possible experience every day and take every aspect of the Dasher’s role seriously. Pay is obviously a big part of the Dasher equation. In this spirit, over the past few months we’ve slowly been rolling out an updated pay model for Dashers: one that is designed to make pay for every delivery more fair and more consistent, while providing more information about the delivery ahead of time. After months of testing and thanks to feedback from thousands of Dashers, the vast majority of Dashers are already on the new pay model and it will be enabled for the few remaining Dashers by the end of the month.
We’re excited about this new pay model for a number of reasons. First, Dashers now earn more for deliveries that are more complex. As a result, over time, our tests showed that more Dashers earn more money under this new pay model. In fact, our data showed that average Dasher pay increased 24% for deliveries with subtotals less than $20 and average Dasher pay increased by about 15% for deliveries that took more than 35 minutes to deliver.
Second, the new pay model offers more clarity around total pay. Dashers now not only know how much money they’ll make for each delivery before they accept it, but they will also see the total estimated driving distance on a map of their route. With these key pieces of delivery information presented ahead of time, Dashers are free to accept or reject any opportunity with more complete information about how much they will earn.
Finally, and perhaps most importantly, we’ve found that most Dashers prefer the new model. The feedback from thousands of survey respondents across the country shows that the vast majority of Dashers like or love the new model. And we’re confident that we’re taking the right approach, after hearing feedback such as this:
“It’s fairer, and I am making more money per delivery on average.” — Orange County Dasher
“The transparency is great and it lets you know exactly what to expect.” — Dallas Dasher
“With the new model, I feel like I’m truly earning the equivalent of how hard I work.” — Washington D.C. Dasher
“I like knowing upfront what I will make and the new model seems to pay better based off distance which makes it worth it for longer trips.” — Atlanta Dasher
So how does the new pay model work? With the new approach, we’re doing away with flat delivery pay, which was the same no matter the size, distance, or complexity of the order. Now, Dashers’ earnings will be calculated using a variable “pay boost” based on, among other things, the complexity of the delivery. Our algorithm determines the complexity based on a variety of factors, including expected driving distance, traffic, parking, size of the order, and whether Dashers have to place the order at the counter. Each market has a minimum floor that Dashers will make on every delivery no matter the complexity ($6.00 in most cases) and Dashers will always see the guaranteed amount they’ll receive for each delivery upfront. And of course, Dashers still receive 100% of tips from customers.
We’ll continue surveying the Dasher community and monitoring feedback in the weeks and months to come, and will use this information to keep improving Dasher pay and the dashing experience as a whole. We’re hopeful that a fairer and more consistent Dasher pay model will encourage those considering becoming a Dasher to jump on board. If you’re thinking about joining the Dasher community — or if you haven’t yet done a delivery with DoorDash — sign up today.
By Richard Hwang and Aamir Manasawala, Software Engineers
One of our goals at DoorDash is to surface to consumers a wide range of stores that are quickly deliverable to their given address. This process involves calculating accurate road distances for each store-consumer pair in our real-time search pipeline. Our earlier blog post about recommendations for search primarily focuses on the ranking component of search at DoorDash. This blog post describes how we architected our search system using open source technologies to help determine consumer selection.
Problem and Motivation
Calculating accurate driving distance in real time is critical to the selection that a DoorDash consumer sees. A mere straight line circle-based distance could be inaccurate and would result in very long Dasher drive times, especially when the topology of the region has unevenness due to barriers like mountains, lakes, bridges, parks, etc.
Figure 1 depicts a circle with a nine mile radius centered around an address in Southern California. If a consumer orders from a store on the edge of this circle, it will take at least half an hour (as shown in Figure 2) just for the Dasher to get from the store to the consumer.
Figure 1 (left): Circle with a nine mile radius centered around an address. Figure 2 (right): Time to drive from point on edge of that circle to given address.
Basic Definitions
Before we delve into the system architecture, let us define some terms:
Latitude, Longitude: A unique location point on the planet, abbreviated as (lat, lng.)
Geohash: A hierarchical encoding system to subdivide space into grid like structure.
Isochrone: A curve of equal travel time, represented as a GeoJSON. Figure 3 shows an isochrone in San Francisco depicting areas that can be reached in 10 (inner region) and 20 (outer region) minutes by walking. Figure 4 is geojson representation of an isochrone.
Figure 3 (left): Isochrones of 10 and 20 minutes (walking). Figure 4 (right): geojson representation of an isochrone.
Architecture
The following diagram describes the overall architecture to determine if a store is in the consumer’s delivery address to determine its selection.
Figure 5: Structure of architecture to determine stores within a consumer’s delivery address.
Offline component:
The offline component involves an isochrone service responsible for computing isochrones for a given location (lat and lng, which is converted to a level seven geohash) and parameters (eg: travel time).
To compute isochrones, we use our custom fork of Galton, an open source project. Galton is built on top of OSRM, an open source routing engine, and concaveman, a fast implementation of a concave hull algorithm. Galton first generates a grid of coordinates of configurable size and granularity around the input coordinate. OSRM then computes travel times from the input coordinate to each of the grid coordinates. Grid coordinates with travel times greater than the input travel time are filtered out. Finally, the concave hull algorithm generates an outline of the remaining coordinates, producing the appropriate isochrone as shown in Figure 6, which is the nine mile isochrone for the same address in Figure 1.
Figure 6: Nine mile isochrone for address in Figure 1.
The service caches isochrones in DynamoDB, as simple key-value lookups for speedy retrieval. Further, we key by geohash, precision 7, rather than exact coordinate, to reduce the number of entries we need to store. Precision 7 geohashes have an error of ±0.076 km; isochrones for coordinates within these bounds will not vary drastically. We store isochrones in order of millions and with lookup time under ten milliseconds.
For each request, the service queries for DynamoDB: if the isochrone is present then it is returned. If the isochrone is absent then an asynchronous job is launched to generate and store it, returning a null response. On subsequent requests for that address and parameters, the generated isochrone will be returned. When we launch a new market, we bootstrap it by running a script to pre populate isochrone entries for all geohashes in the market, to get the market up to speed for accurate selection upfront.
Online component:
DoorDash clients call the search backend API for the specific (lat, lng)
Search module calls isochrone service with the (lat, lng) and parameters like travel time to fetch the corresponding isochrone. These parameters are district-specific and configurable, so we can run experiments for testing conversion changes based on selection. If the isochrone is absent (as in the case when the isochrone is absent in dynamodb), we fall back to the naive straight line distance computations (with tighter radius). We persist the selection logic (isochrone or straight line along with parameters) at a session level in the backend search module to provide a consistent notion of selection across browsing sessions for the consumer.
The search module on fetching the isochrone for that address is encoded as a polygon geoshape to construct a geoshape query to hit Elasticsearch.
Stores that are indexed into Elasticsearch have the store location encoded in geo-point format. Elasticsearch builds a prefix tree structure at index time to support fast geo queries at runtime. Elasticsearch runs the given ES geoshape query from Step 3 to compute an intersection of the polygon with stores in the index for retrieval.
Store results are deserialized and returned to the client for that address.
Conclusion
Our current implementation accounts for the topology of the region via driving distance addressing the inaccurate selection problem in Figure 1 by isochrone selection as shown in Figure 6. Furthermore, this architecture allows flexibility to configure and control selection logic based on regionality, to dynamically change selection logic based on supply/demand curves, and to run selection experiments.
Some potential areas that we will be working on in the future include getting more accurate real-time traffic and road condition updates into the system.
If you are passionate about solving challenging problems in this space, we are hiring for our Search & Relevance team and our Data Infrastructure team. If you are interested in working on any other areas at DoorDash check out our careers page. Let us know for any comments or suggestions.
Food is meant to be shared. Whether it’s with family, friends, or colleagues, there’s nothing better than sitting around a table and enjoying a meal together.
For more than two years, DoorDash has offered a group ordering feature on the web. Group ordering allowing anyone to start a group order cart and have people add their selected items directly, making it simpler and easier for people organizing big group delivery orders. Over the years we’ve continued to add functionality to this feature, like allowing organizers to set a price limit on each person’s order (great for office managers ordering lunch for a big meeting).
Today we’re continuing to build out group ordering by releasing our most requested feature: group ordering on the DoorDash app. Starting today, DoorDash on Android and iOS now support creating and adding to group orders directly in the apps.
How it Works
Organizers can create a group order on any store menu or order cart. Simply visit the menu of your favorite restaurant and click the “group” icon in the top right. Organizers can then set a subtotal limit per person and will receive a link that they can share out to cart participants.
Once everyone has submitted their requests and the full order has been placed, both participants and organizers can track the status of the delivery through notifications and the live delivery map. On iOS, group ordering is also integrated directly with iMessage, creating a native experience to view and track a delivery when you share a group order through your chat conversation.
DoorDash is the first and only app to offer mobile group orders. So whether you’re planning a meeting at work, hosting friends for the football game, or just having dinner with the family, DoorDash’s mobile group orders make deliveries simpler, easier, and faster than ever.
By Mike Dennis, National Partnerships Marketing Manager
We’re thrilled to announce that we are expanding our partnership with P.F. Chang’s to make their complete bottled beer and wine menu available for delivery. You can now order delivery of more than 55 varieties of bottled beer and wine from select P.F. Chang’s locations in California.
Whether it’s a pinot noir in Palo Alto, a Kirin lager in Long Beach, or an IPA in Irvine, we’re confident you’ll find the perfect pairing alongside your favorite entrée from P.F. Chang’s.
Placing an alcohol delivery order is easy. Simply open the DoorDash app or visit our website, navigate to the PF Chang’s store near you, and you’ll see all the wine and beer selections next to the rest of the P.F. Chang’s menu. Customers must present a valid government-issued photo ID to prove they are above legal drinking age at time of delivery.
Today DoorDash is launching in Edmonton, Alberta Canada and St. Louis, Missouri. With these newest markets, DoorDash enters its fourth market in Canada and 48th market across North America. We’re excited to expand our service to families in these new regions and continue to build our relationships with local and national partners across the U.S. and Canada.
In the St. Louis area, we’re now serving the cities of Creve Coeur, Clayton, Chesterfield, and Maryland Heights. People in these neighborhoods will be able to order from their favorite local restaurants as well as DoorDash’s national partners like California Pizza Kitchen, The Cheesecake Factory, and P.F. Chang’s .
In the Edmonton area we’ll be serving St. Albert, Sherwood Park and everything inside the Henday Ring, including the neighborhoods of Downtown Edmonton, Strathcona, Westmount, Beverly, Silver Berry, Bonnie Doon, Ottewell, Balwin, Eaux Claires, Pembina, Glenwood, Blue Quill, Royal Gardens, Jasper Place, Pleasantview, Garneau, Inglewood, Terra Losa, La Perle, and Summerlea. Delivery from top restaurants are just a tap away through our partnerships with Rostizado, Leva, Pasta Pantry, Sherlock Holmes Pubs, and Oodle Noodle.
Churros from Rostizado in Edmonton
To celebrate the launch, we’re offering customers in Edmonton, St. Albert and Sherwood Park free delivery for the next three months, as well as some freebies like free churros and pasta. On Thursday, October 5th from 2–4pm, DoorDash is observing “Churrsday” by giving away 200 lunch-sized churro desserts from Rostizado (limit one per customer). The celebration continues on Sunday, October 15 from 2–4pm when DoorDash is giving away free pasta from Pasta Pantry.
Meanwhile, those in the St. Louis area can also join in the fun, with $5 off orders of $15 or more with the code DASHSTL.
With DoorDash, a delicious meal at your doorstep is just a few taps away. If you know folks in these newly launched areas, you’re also in luck — refer them to try DoorDash and you both get a $7 referral credit.
By Brandon Silverman, Senior Strategy & Operations Manager
As Las Vegas has been working to recover from the events of October 1, those of us here on the DoorDash Las Vegas team have been working to support the community in our own way.
Together with our restaurant partners, we are delivering meal donations to victims, first responders and others that have been impacted by this tragedy. For example, this week we teamed up with Streets of New York Pizza to donate meals to more than 1,000 families and volunteers at the Family Assistance Center in the Las Vegas Convention Center. We’re also working with Opportunity Village to identify other communities that may need food and other supplies.
We’ve set aside resources to continue to support the community with meal donations. If you or your organization is in need of meal donations, please reach out to us directly at lasvegasstrong@doordash.com. If you’re looking for other ways to help, please visit the Las Vegas Victims Fund.
From all of us on the DoorDash Las Vegas team, we’d like to give a big thank you to Opportunity Village and Streets of New York Pizza for your help in the relief efforts so far. And to the Las Vegas community of Dashers, customers, restaurants and everyone else in the city, we stand #VegasStrong with all of you.
Today DoorDash is launching in the Salt Lake City area, bringing customers local favorites from hundreds of restaurants. We’re thrilled to add Salt Lake City to our growing delivery footprint, marking our 49th market in North America.
In the Salt Lake City area, we’re now offering delivery in Downtown, Capitol Hill, Agricultural Park, South Salt Lake, White City, Bluffdale, Murray, Midvale, Cottonwood Heights, Sandy, Draper, Riverton, South Jordan, West Jordan, Taylorsville, West Valley City, Herriman, Magna, Holladay, Millcreek, Sugarhouse, and Kearns.
In celebration of this new expansion, customers in the Salt Lake City area will receive their first delivery free and $5 off orders of $15 or more with promo code DDSLC.
DoorDash provides a fast and easy way to get a high-quality meal delivered right to your doorstep. If you have friends or family in Salt Lake City, refer them to try DoorDash and you both get a $5 credit on us. Enjoy!
Today we’re launching DoorDash in Baltimore, our 50th North American major metropolitan market. Delivery is available across Baltimore and surrounding neighborhoods, including Columbia, Ellicott City, Arbutus, Catonsville, Halethorpe, Hanover, and Elkridge.
We’re proud to be expanding to Baltimore, after more than two years serving Washington DC and southern Maryland. Customers can order from hundreds of national restaurants and local Baltimore favorites including:
As a special treat for our launch, we’re partnering with Sticky Rice to give away free sushi burritos during lunchtime on October 25. But if you miss the burrito giveaway there are still plenty of other ways to celebrate. For the next month, Charm City customers will receive free delivery on their first order. Plus, you’ll receive $5 off your first order of $15 or more with the promo code BALT.
Check out DoorDash today and see everything that Baltimore restaurants have to offer.
Over the past eleven days, we have been closely following the Northern California wildfires and we are heartbroken for those impacted in our community.
As a Bay Area-based company, this truly hits close to home, and we are using our platform to do our part to help with relief efforts. Last week, we began working with Noah’s Bagels and West Brooklyn Pizza to deliver food to local shelters. With their help and generosity, we have helped deliver meals to evacuees and volunteers based at San Geronimo Valley Community Center and Lawrence Cook Middle School in Santa Rosa. Since then, we’ve been in touch with nearly two dozen other organizations and we are working closely with them to determine the best ways to support their needs.
If you are familiar with any organizations in need of food deliveries, please complete this form to let us know how we can help.
DoorDash has also donated to the American Red Cross, which is coordinating relief efforts across the region. You can join us in supporting those affected by the wildfires by donating directly to the American Red Cross.
You can’t properly ring in the New Year without some bubbly, and thanks to DoorDash’s alcohol delivery in select cities, you won’t need to leave the house to stock up your bar.
Follow my easy steps below and learn how to make a super simple sparkling cocktail that is sure to please any crowd. It requires just three ingredients and elegantly straddles the line between sweet and tart.
You will need:
1. Elderflower liqueur: Made from the flowers of the elderberry tree, this adds amazing aromatics and the perfect sweetness to the cocktail. Splurge on this, thank me later.
2. Fresh grapefruit juice: The bittersweet juice is the perfect balance to the sweetness of the elderflower liqueur. In this cocktail, it adds some color as well.
3. Sparkling wine: I went with a dry Prosecco to avoid an overly sweet cocktail, but almost any sparkling wine will suffice. You don’t have to spend a fortune either, this was under $15 per bottle.
Instructions:
· Add 1 oz Elderflower liqueur to a flute or coupe glass
· Add 1 oz grapefruit juice
· Top with very cold sparkling wine
· Garnish with a twist (optional)
Bonus: I’m a fan of batched cocktails freeing up the host to interact with guests, instead of making drinks, and this cocktail is no different. You can batch the elderflower and grapefruit juice ahead of time by placing equal parts in an air tight container and storing in the fridge. When it comes time to serve, just pour in 2 ounces of your batched mix and top with sparkling wine. I’d advise batching no more than 24 hours in advance since it’s fresh fruit juice.
Last but not least: we’re giving away $100 in DoorDash credits plus a $100 Essential Cocktail Kit from The Barrel Aged Dad. To enter, find us on Instagram and look for the official entry post! Click here for terms and conditions.
Cheers and Happy New Year!
A Cocktail to End The Year Right was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
DoorDash is now available in Kansas City, adding another major metropolitan area to our Midwest markets.
We’re excited to serve customers in Downtown Kansas City, Midtown, Overland Park, and on January 17 will expand delivery to food lovers in Independence, Blue Springs, Platte County, and Clay County. Delivery is available from 11 a.m –10 p.m from more than 1,500 restaurants in the area.
In celebration of our newest market, KC area customers can use promo code DASHKC for $5 off orders and enjoy $.99 delivery for the first week.
DoorDash is a fast, easy and reliable way to get your favorite restaurants delivered to your doorstep. If you have friends or family in the Kansas City area refer them to try DoorDash and you both get a $10 credit on us. Enjoy!
Now Dashing in Kansas City was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
..and eventually to all our national partners in the area, such as Buffalo Wild Wings, P.F. Chang’s, Red Robin and BJ’s. Delivery is available 10:30 a.m — 11:30 p.m.
To get everyone excited for our launch in Tucson, enjoy your first DoorDash delivery on us. Also, get $7 off your first order of $15 or more using code TUCSONLAUNCH, and for the first month, all deliveries will be $1.99 or less.
DoorDash is a fast, easy and reliable way to get your favorite restaurants delivered to your doorstep. If you have friends or family in the Tucson area refer them to try DoorDash and you both get a $10 credit on us. Enjoy!
Now Dashing in Tucson was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
By David Kastelman, Business Operations Manager (SF), Andrew Susanto, Operations Associate (LA), Elle Pak, Strategy and Operations Manager (NY)
We’re thrilled to kick off Project DASH (DoorDash Acts for Sustainability and Hunger), our new initiative focused on tackling the problems of hunger and food waste in the local communities we serve. The average restaurant has 100,000 pounds of excess food every year¹ but only 1.4% of that is donated; 78% of restaurants cite transportation as a barrier to donating more food, making last-mile logistics the most frequently cited roadblock to increased food rescue.² Leveraging DoorDash’s passionate employees, last-mile logistics technology, and a network of millions of Consumers, hundreds of thousands of Dashers, and tens of thousands of Restaurant Partners across more than 600 cities, Project DASH consists of the following:
1:1 Meal Donation: DoorDash will donate one meal through the Feeding America® Charity for every order placed on DoorDash’s website, Android, or iOS app through the month of January.³ To amplify this effort, DoorDash will leverage the company’s national merchant footprint and invite one Restaurant Partner each month to join us in the “one-meal for one-meal” initiative in 2018, with plans to extend further in the future. One in eight Americans faces hunger, and this meal match will ensure that Feeding America® is able to help more families and individuals put food on their tables.⁴
WeDash For Good: Since DoorDash launched four and a half years ago, employees have “dashed” each month in an effort to keep employees connected to the experience of Customers, Dashers, and Restaurant Partners. Moving forward, all funds earned in this effort will be donated to Feeding America® to further our partnership.
Reducing Food Waste: DoorDash will introduce a pilot program that pairs the company’s logistical expertise with Feeding America® Charity’s MealConnect system. MealConnect serves as a hub for food donations, linking surplus food with local nonprofits that can use it to feed those in need. DoorDash will use one of its core competencies — specifically DoorDash Drive, the enterprise-fulfillment arm of the business — to help bring surplus food to local food banks through the MealConnect system. The integration of DoorDash Drive with Feeding America®⁶ Charity’s MealConnect program brings together the nation’s foremost platform for fulfilling on-demand delivery requests with the nation’s largest domestic hunger-relief organization. A special thanks to the DoorDash employees who began piloting food donations at the August 2017 Hackathon⁵: Christina Zhang, Dawn Lu, Kevin Fu, Joseph Oh, David Emanuel, Jen Zink, Rachel Marincola, Hilary Brown, Usman Gul, and Bret Hamilton. A debt of gratitude to the restaurants and food pantries that have so far participated in piloting food donation including: Bodhi Bowl, Eataly, KaynDaves, BiiBiip Mediterranean, Feast From The East, Veggie Grill, The Midnight Mission, Westside Food Bank, World Harvest Food Bank, and the Sylvia Rivera Soup Kitchen. And, as always, a huge thank you to the Dashers who have fulfilled these important deliveries and who make it all happen on the DoorDash platform each and every day.
Please join the effort! Are you a restaurant interested in donating surplus food? Email projectdash@doordash.com for more information.
⁵ Find out more about what a Hackathon is or how they work at DoorDash here.
⁶Feeding America® is the largest hunger-relief organization in the United States. Through a network of 200 food banks and 60,000 food pantries and meal programs, we provide meals to more than 46 million people each year. Feeding America also supports programs that prevent food waste and improve food security among the people we serve; educates the public about the problem of hunger; and advocates for legislation that protects people from going hungry. Individuals, charities, businesses and government all have a role in ending hunger. Donate. Volunteer. Advocate. Educate. Together we can solve hunger. Visit www.feedingamerica.org, find us on Facebook or follow us on Twitter.
Introducing Project DASH was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
When we launched DoorDash in 2013, things were pretty different. Back then, we were called “Palo Alto Delivery” and had just eight menus to choose from.
But I’ve always believed in the potential of DoorDash and what we stood for. And while we’ve spent the last four years — heads down — working hard to build our business and improve our service, we’ve never shared our story. Until today.
My co-founders and I started DoorDash to help local businesses. Growing up in the Midwest, I watched my mom work tirelessly in her restaurant, where I worked alongside her to help achieve her American Dream. She ultimately saved enough money from her restaurant to start a medical clinic and return to her roots as a doctor.
I see my mom’s story everywhere these days. Everyone is passionate about fulfilling their own American Dream. Whether you’re a working mom looking to make a difference at work and home, a college student seeking flexible ways to pay tuition, or a restaurant owner focused on growing your business, we salute you and the grit, ambition, and resourcefulness you show daily. At DoorDash, we stand for people like you — the everyday person constantly looking to improve in order to realize their full potential.
Our mission is to deliver good by connecting people and possibility. For Consumers, we help you spend more time with your friends and family, or get ahead on your favorite projects. For Dashers, we offer flexible work opportunities to help you achieve even bigger goals. For Merchants, we increase your reach and help grow your business.
To help tell our story, we’re launching a new set of initiatives to deliver good within our communities.
You’ll see a brand new logo and a visual identity that reflects the spirit of DoorDash. With the refreshed logo, we’re reinforcing all the qualities the company has always thrived on: making bold moves, being relentless and fast-moving, standing for reliability and efficiency.
During the first wave of our “Delivering Good” movement, for every order placed with DoorDash from now until the end of January, we’ll donate one meal in partnership¹ with Feeding America®².
Finally, we’re kicking off Project DASH, our new initiative focused on tackling the massive problem of food waste and hunger. The average restaurant has 100,000 pounds of excess food every year, and only 1.4% of it is donated. The biggest barrier to donation, according to 78% of restaurants? Not having the logistical means for delivering to food banks and shelters. To help, we’ve rolled out programs in New York, Los Angeles, and the Bay Area, and there will be more to come in 2018.
I want to personally thank you for making DoorDash the success it is today — we wouldn’t be here without you. We are humbled to be part of your journey and what it represents. And I hope you will join us in the movement to deliver good.
Tell us how we help “deliver good” into your life! If you’re up for it, please email your story to me at tx@doordash.com — I’d love to hear your experiences, and I know the rest of the DoorDash community would, too.
Upward and onward!
¹$1 helps provide at least 10 meals secured by Feeding America® on behalf of local member food banks.
²About Feeding America: Feeding America® is the largest hunger-relief organization in the United States. Through a network of 200 food banks and 60,000 food pantries and meal programs, we provide meals to more than 46 million people each year. Feeding America also supports programs that prevent food waste and improve food security among the people we serve; educates the public about the problem of hunger; and advocates for legislation that protects people from going hungry. Individuals, charities, businesses and government all have a role in ending hunger. Donate. Volunteer. Advocate. Educate. Together we can solve hunger. Visit www.feedingamerica.org, find us on Facebook or follow us on Twitter
Delivering Good: It’s a Movement was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
What an exciting week to be an Eagles fan! On the heels of Sunday night’s big win, DoorDash is thrilled to announce that today we are launching in Downtown Philadelphia!
With today’s announcement, we’ll now be available across the Greater Philadelphia Area, including University City, Center City, and along the Main Line, from Spruce Hill in West Philadelphia to Penn’s Landing.
Residents of Downtown Philadelphia will now be able to enjoy door-to-door delivery from DoorDash’s local and national partners, including Wendy’s, The Cheesecake Factory, McCormick and Schmick’s, Halal Guys, and Chick-fil-A, and local favorites, like New Delhi Indian Restaurant, Bleu Sushi, Monsu, and Al Zaytouna.
To celebrate our Philadelphia expansion, customers can use promo code PHILLYDELIVERS for $5 off orders of $15 or more.
DoorDash is a fast, easy and reliable way to get your favorite restaurants delivered to your doorstep. To search DoorDash for local favorites or to discover your next go-to, visit doordash.com or download DoorDash for Android or iOS.
Enjoy!
DoorDash Arrives in Downtown Philly was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
Starting today, DoorDash will be available in Westlake, Bay Village, Rocky River, North Olmsted, Cleveland Heights, University Heights, Shaker Heights, Mayfield Heights, Warrensville Heights, and Cuyahoga County.
To celebrate the debut of DoorDash in Cleveland, customers can use promo code DASHCLEVELAND for $5 off an order over $15, valid throughout February.
To search DoorDash for local favorites or to discover your next go-to, visit doordash.com or download DoorDash for Android or iOS.
Now Dashing in Cleveland was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
We will be immediately available in Novi, Farmington Hills, Birmingham, Clawson, Troy, Rochester Hills, Sterling Heights, Mount Clemens, Warren, and Oakland and Macomb counties, with plans to expand further into downtown Detroit later this month. Customers in the Detroit area can order DoorDash between the hours of 10 a.m. and 10 p.m.
“It’s hard to find a partner who sends us orders and enable us to complete the deliveries, but now DoorDash is covering us on both fronts,” said Gil Stebbins, CFO of Renee’s Gourmet Pizzeria. “The on-boarding process was easy, so that’s a great sign for what’s to come with this partnership.”
To celebrate the debut of DoorDash in Detroit, customers can use promo code DASHDETROIT for $5 off an order over $15, valid throughout February.
To search DoorDash for local favorites or to discover your next go-to, visit doordash.com or download DoorDash for Android or iOS.
Now Dashing in Detroit was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
Any way you slice it (or fold it, roll it, or if you really wanna get fancy, dish it) pizza, the ultimate order for delivery, gets a lot of love at DoorDash.
Today is National Pizza Day, so in our grand tradition of pizza by the numbers, we thought we’d see which city’s passion for cheesy pies reigned supreme this time around. Drumroll, please…
San Francisco Bay Area beat Atlanta, the #1 city last fall, for most pizza ordered.
MOST ALL-TIME PIZZA ORDERS
San Francisco Bay Area
Los Angeles
Orange County
Chicago
Dallas
Houston
Phoenix
Boston
San Diego
Atlanta
So what kind of pizza tops everyone’s list when ordering? Let’s just say mozzarella is very, very popular.
MOST POPULAR PIZZA TYPE BY CITY
San Francisco Bay Area — Margherita
Los Angeles — Margherita
Orange County — Pepperoni
Chicago — Cheese
Dallas — Cheese
Houston — Cheese
Phoenix — Cheese
Boston — Cheese
San Diego — Pepperoni
Atlanta — Build Your Own
More fun facts:
In 2017, December was by far the most popular month to order pizza — but Halloween weekend also topped our list. So many holiday parties, so many pizzas.
MOST POPULAR DAYS FOR PIZZA IN 2017
Saturday, December 9
Saturday, December 16
Sunday, October 29
Saturday, December 2
Sunday, December 17
And the biggest single order for pizza? A whopping 60 large pizzas from a Little Caesars in Arizona on April 17 — the day before Tax Day, by the way. We’re guessing that fed a lot of hungry accountants that night.
So, grab a slice (or several) and celebrate our favorite cheesy holiday. What toppings are your favorites?
No pizzas were harmed in the making of this GIF.
10 Cities for Pizza Lovers was originally published in DoorDash on Medium, where people are continuing the conversation by highlighting and responding to this story.
David Kastelman, Data Scientist & Raghav Ramesh, Machine Learning Engineer
To A/B or not to A/B, that is the question
Overview
On the Dispatch team at DoorDash, we use simulation, empirical observation, and experimentation to make progress towards our goals; however, given the systemic nature of many of our products, simple A/B tests are often ineffective due to network effects. To be able to experiment in the face of network effects, we use a technique known as switchback testing, where we switch back and forth between treatment and control in particular regions over time. This approach resembles A/B tests in many ways, but requires certain adjustments to the analysis.
Dispatch at DoorDash
The core responsibility of the Dispatch system at DoorDash is to power fulfillment of every delivery in Doordash’s three-sided marketplace of Consumers, Dashers and Merchants. To effectively achieve this, we focus on
the core assignment problem — deciding which dasher is the best suited to fulfill a delivery, in an efficient, robust and scalable way.
machine learning algorithms to predict the numerous timepoints in the life of a delivery — “when can this order reach the consumer”, “when will this order order be ready for pickup?”, etc.
algorithms to decide how and when to group multiple deliveries headed in similar directions at similar times
how to leverage various marketplace shaping strategies to balance supply and demand, including pricing changes
Illustration of the core assignment problem
As we continuously iterate on these problems, we rely heavily on both simulation and experimentation to best serve our audiences on all three sides of the marketplace. Offline simulation is helpful for checking assignment algorithms, and offline evaluation and A/B testing helps us with iteration on prediction algorithms. However, it is no surprise that in dealing with marketplace problems online, network effects often make traditional A/B testing ineffectual. We’ll explain that with an example of SOS pricing below.
Marketplace Experimentation
SOS pricing is a demand shaping strategy used when we have too few dashers compared to the incoming delivery volume. In these scenarios, in order to avoid overwhelming the Dasher fleet with an intractable number of deliveries, thereby increasing Consumer delivery times, we enable SOS pricing, which increases delivery fee. With this increase, demand gets throttled and shifted to later times; meanwhile, dashers are motivated to dash more. When we introduce (or later, modify) the SOS pricing algorithm, we would like to experiment to understand the impact on customer retention and delivery times. If we were to test changes as an A/B experiment on Consumers, 50% of Consumers would see SOS pricing and 50% wouldn’t. In this case, the first set of Consumers get only half of the benefit of supply equilibration (by extension, reduced impact on Consumer delivery times), while the other half get the partial benefit without any extra pay, which makes our learning incomplete. The problem here is the network effect — both sets of Consumers share the same Dasher fleet — so adding a Consumer to the treatment group also affects the experience of Consumers in the control group and we can not establish independence between the two groups.
One way to get around network effects is to introduce the change and observe the impact before and after. For instance, we can compare Consumer delivery times before and after the introduction of SOS pricing. The main problem with this approach is the famous maxim that correlation doesn’t imply causation. Without a randomized experiment, we cannot be sure if the results we are seeing after our change are really because of that change, or just coincide with other things changing in the system. DoorDash is a dynamic company with a lot changing every day, so relying on a pre-post comparison is a risky proposition. At a minimum, relying on pre-post analyses would cause a dispatch bottleneck where we could only change one big thing at a time. Even this strategy, however, is insufficient for interpreting correlations as causal because occurrences outside dispatch’s control like consumer promotions and weather have a big impact on our normal dispatch metrics and might be the true cause of a pre-post observed difference. To take the most-cited line from the oft-cited Airbnb medium post on A/B testing, “the outside world often has a much larger effect on metrics than product changes do.”
Switchback Experimentation
Due to the limitations of A/B tests and the insufficiency of pre-post comparisons, the Dispatch team recently decided to switch to a new analytic framework for much of its experimentation — ‘switchback testing.’ Fun fact: switchback testing was originally employed in an agricultural context, specifically for cow lactation experiments. MOOOOO.
In switchback testing, the core concept is that we switch back and forth between control and treatment algorithms in a certain region at alternating time periods. For example, in the SOS pricing example, we switch back and forth every 30 minutes between having SOS pricing and not having SOS pricing. We then compare the customer experience and marketplace efficiency between the control time bucket and treatment time bucket metrics corresponding to the decisions made by the algorithm during the two periods.
Implementing Switchback Experiments
In implementing switchback experiments, we include two extra pieces of logic:
Illustration of time-market unit randomization
We randomize the variant used for each time window, rather than simply randomizing the initial variant and flipping back and forth deterministically. With this approach, each time unit is a randomized experimental unit.
We further split across different geographical units (regions) and randomize them independently, so region A could use the current algorithm for one window and the new algorithm for the next, while region B could use the new algorithm for the first window and the old algorithm for the next. This in a sense actually creates a series of ‘time-region units.’
In many ways, switchback testing is exactly like A/B testing, but instead of randomizing based on something like deliveries, we randomize based on time-region units.
The rest of the section explains the design of the switchback service we use at DoorDash.
The switchback service is responsible for three functions
Storing metadata on all experiments (Metadata)
Choosing the algorithm variant to use at certain point of time for any experiment (Bucketing)
Handling tracking to enable metrics calculation and analysis (Tracking)
The most important metadata of an experiment is the “switchback window”, i.e., the duration of time for which we persist a variant before switching. Other metadata include names and relative sizes of the bucket variants. Internally, the switchback service stores the following state — 1. The start of the current time window, 2. A map with region id as the key and the current bucket name as its value and 3. A unique identifier for the experiment unit (time + region)
To achieve bucketing and tracking, the switchback service exposes two endpoints
Registration
Based on the time passed by the registration endpoint, the switchback service checks and updates its internal state e.g. check if it is time to update the buckets and if so, flip a (multi-sided) coin for every region and determine the bucket for the new time window
GetBucket
Used by the experimenter system to ask for the current bucket (variant) for an experiment
the switchback service fetches the bucket value from its state and sends a tracking event (which includes experiment metadata and unit identifier) to the ingest pipeline.
A set of ETL jobs that combine the above tracking events and our metrics tables enable analysis of the experiment results.
Analyzing Switchback Experiments
In order to use something like a t-test (which is often used to analyze the results of A/B tests) for our switchback experiments, we analyze based on the same units we’re randomizing on: namely time-region units. This means we’re conducting our statistical tests on the average values of control and treatment time-region units, rather than the individualvalues of control and treatment deliveries. Therefore, the average value for the treatment and control groups is a simple average of all time-region units rather than a weighted average of those values (weighted by deliveries). This is illustrated with a simple example of 14 deliveries below. While the simple and weighted average of values usually converge over time, if they do not, it is an indication that your intervention has a different impact on units with a lot of deliveries than it does on units with fewer deliveries. In the future, we plan to handle these divergent situations by using multi-level modeling (MLM, also known as hierarchical modeling). MLM also has the advantage of likely being able to reduce the margin of error of our statistical tests.
Sample of 14 deliveries (numeric table in appendix) with illustration of weighted vs. simple average
Moreover, traditional A/B testing has an assumption of independent units, which means that if we know something about the behavior of one unit, it does not tell us anything about the behavior of the second. This assumption is clearly violated in the case of time-region units as the performance of one time-region is highly correlated and can influence the performance of the next. To give a more concrete example, the average time it takes to complete deliveries in one area in one 10-minute chunk of time is related and highly correlated to the average time it takes to complete deliveries in the same area in the next 10-minute chunk of time — much more so than in the case of a single delivery and the next delivery assigned. To account for this lack of independence in our statistical tests, we use an approach robust to a lack of independence called a sandwich estimator of variance (R-snippet of the code we use included in appendix). While strictly speaking this approach (or something similar) is required to avoid violating core assumptions, thus far when running A/A tests (a dummy experiment where there’s no actual difference between “treatment” and “control”) we’ve found the sandwich estimator’s effect on our variance calculations has been <10%.
Setting up Experiments and Doing Power Calculations
In creating our time-region units and determining how granular the geographic units should be and how quickly to switch back and forth, we really have to consider two factors: bias and margin of error.
Bias occurs when your randomization of time-region units is compromised, and the sorts of deliveries that are in treatment and control group are not the same on average. For instance, we might test a certain algorithm which is reluctant to assign deliveries with longer distances between stores and customers. As a result, this treatment algorithm may be observed to have a faster average time to completion even if it is not a better algorithm simply because it is cherry-picking short deliveries and leaving the control group to complete longer deliveries. Bias is more likely to occur if your regions get too small or you switch back too frequently; when we switchback less frequently, it forces the treatment group to complete many more of the longer deliveries and rely less on the control group to clean up its mess. More on health checks to guard against bias and check successful randomization in the footnote.
Margin of error is how much uncertainty exists in our estimate of the impact of an intervention. There are two main factors that influence the uncertainty in our estimate: the natural variation in the data we are observing and the number of units in our dataset. Without getting too technical (refer to sampling distributions for more information), the margin of error in our estimates is proportional to the natural variation in our sample divided by the square root of the number of samples. As time-region buckets get more granular, natural variation in a metric tends to go up, however more granular time-region buckets provide more units in our dataset which drives down uncertainty.
To understand the interplay between natural variation and number of samples, we ran a series of long-term A/A tests. These tests allowed us to look at how margin of error differed for the average value of a certain metric in a time-region unit as we switched back and forth more quickly or more slowly. The results for one metric are summarized in the table below (note, given the small impact of the sandwich estimator in most situations, it is ignored in this table):
We were able to do a similar sort of analysis using different levels of geographic granularity.
Summarizing the following above considerations about bias and margin of error: we do not want time-region units that are too small or we risk introducing bias and failing one of our randomization health checks; on the other hand, we do not want time-region units that are too large or we risk having large margin of errors and taking too long to get our learnings. Based on these analyses, we typically run our tests with 30 minute switchback periods using certain geographic divisions roughly at the city-level.
Validation of our Switchback System
An extra verification step we are interested in is “are the results from a switchback experiment reflective of reality?”. Concretely, if we see an X% increase in a metric in an experiment and we completely move over to the new algorithm, will we actually observe the gains in the metric in the real world? This is particularly important from a business perspective and for making reliable product decisions based on experimental results.
To do this, we observe the changes in metrics prior to and after the launch of the new algorithm. As described earlier, observing changes in a pre/post scenario is challenging for multiple reasons. However, over large time windows after product launches, we can reliably observe the impact on the metrics time series. So, what we look for here is just directional confirmation of the results we observe in the experiment.
Conclusion
As DoorDash tackled this problem, we were able to find some helpful resources like Lyft’s 3-part discussion of experimentation in a ridesharing marketplace. We felt there were still some questions left to answer and believe we’ve been able to make some additional observations particularly on the analysis and implementation details, and we hope our observations may help you if you’re considering similar issues. That being said, we fully admit we’re just getting started on unpacking exciting challenges like the ones discussed above. Special thanks to Yixin Tang and Viraj Bindra for their help in publishing this post.
See something we did wrong or could do better, please let us know! And if you find these problems interesting, come work with us!
Appendix
R-Snippet for Analysis with Sandwich Variance Estimator
Numeric Data for Sample of 14 deliveries used in example in article above
¹ To guard against bias and ensure ‘successful randomization’, you can check to make sure that factors that are unaffected by your intervention have equal values between the treatment and control group: in our example, a good value to check would be the distance between restaurant and customer. At DoorDash, however, we often find that simply checking to make sure the expected proportion of deliveries are considered by treatment runs and control runs sufficiently guards against most instances of bias. To return to our example of an algorithm that is hesitant to assign long-distance deliveries, this would result in unequal proportion of deliveries being assigned the control group rather than the treatment group and this deviation would indicate the existence of bias.
² For example, average delivery times in a specific 5-minute window at a city level varies more widely than can average delivery times across all of DoorDash’s regions in an hour.
³ However, if we notice any bias, we shut down the experiment and start over with coarser time and/or geographic units.
