CASE STUDY
How Lights Online increased CVR |READ NOW
Keeping Catalog Models Up-to-Date with Shopify Webhooks
By
Chris Patrick
December 15, 2021

At Skafos, we help Shopify merchants’ customers discover their perfect products, using a mix of fancy machine learning models built upon different product characteristics like images, titles, and descriptions. But a shop’s catalog can change from day to day: shops can add products, delete products, change the images for products, sell out of a particular product, or get more inventory for a product that was previously sold out.

So how does Skafos keep models of a shop’s catalog as up-to-date as possible, ensuring the best possible recommendations?

________

Polling vs. Listening

Shopify offers many tools to help app developers and shop owners manage their stores, including support for both polling and listening for updates. In the context of Shopify development, here’s what we mean when we talk about polling and listening:

Polling — requesting information from Shopify about an entity of a set of entities, such as a set of products in a shop’s collection

Listening — receiving information from Shopify about particular events (such as the addition, update, or removal of a product)

Here at Skafos, we take advantage of BOTH of these approaches to ensure our model of a shop’s catalog is baked with fresh-out-of-the-oven goodness.

Why do we use both approaches? Polling for updates allows us to capture the addition or removal of any products at the point in time when we’re ready to train our machine learning models, in addition to making sure our representation of each product is current. However, information about products can change between trainings; for example, a product’s price may change because a great new sale has just started. Listening for updates, and subsequently updating our representation of that product in our database, allows us to accurately display information about that product to the shopper (for example, showing that new fantastic sales price) when we make our recommendations.

How does polling and listening work with Shopify?

Polling

Let’s first talk about polling. With an active Shopify access token for a particular shop, you can make queries to Shopify using GraphQL, as in this query grabbing the id, title, and product display page link for a shop’s first 100 products:

{
 products(first: 100) {
   edges {
     node {
       id
       title
       onlineStorePreviewUrl
     }
   }
 }
}

Shopify offers tons of support for GraphQL and lots of other app and store developer tools as well, making it super easy to get started. When we at Skafos first began working with Shopify, we used basic GraphQL queries to request information about a few products at a time to ensure we didn’t run into problems with rate limiting. However, Shopify also offers the ability to run bulk queries, and we quickly pivoted to using those as they run MUCH more quickly and efficiently when you need to request a lot of information, such as product details on all products for a shop — and no more need to worry about rate-limiting!

Listening

If you want to instead (or additionally…) listen for updates about a shop, a collection, or the shop’s products, you can subscribe to Shopify webhooks. Here at Skafos, we stay current on key information about our shops using 2 webhooks in particular: products/update and orders/updated (not sure why one is “update” and the other is “updated,” but make sure you’re getting those names correct). The nice thing about the update/updated webhooks is that you’ll get information not only when an entity is changed, but also when a new entity is added — that way you don’t have to subscribe to multiple webhooks to get the information you need. In fact, we recommend NOT subscribing to both, say, the products/update and the products/create webhooks, because you’ll get 2 events at the same time upon product creation, and your system is more likely to be overwhelmed.

Some key things to keep in mind about webhooks:

  • You need to be able to respond with a 200 code, letting Shopify know you’ve received the webhook event. If you have issues responding to the webhooks, Shopify will mark the webhook event as failed and make additional attempts to send you the event. Also, if you have enough failures, Shopify will automatically remove your webhook. Don’t worry if this happens to you because it did to us — more than once! — and all you need to do is re-register for the webhook.
  • The update webhooks will not tell you what information about the entity has changed.😢
  • By default, you’ll receive all of the information about the entity, but you can specify the fields you’re interested in receiving when you first register for the webhook.
  • In order to receive webhooks, you’ll need the proper access scopes for a shop, so if you’re an app developer you’ll need to request all of the permissions for your app upon install. At Skafos, we request the write_products (which includes read_products access) and read_orders scopes in order to get access to the webhooks listed above. It’s a bit of a pain for the shops using your app to have to go back and give additional access after an initial install, so think broadly about what scopes you’ll want to request down the road to make sure you have as many of those in place to start with as possible.

You can easily monitor your webhook performance by using Shopify’s dashboards for your app. To find this dashboard:

  • Find your app in the list of apps, then click on the link in the Webhook Deliveries column. Note that the value in this column can be “No Deliveries” both for apps without webhooks as well as for apps that have webhooks with no failures in the past 7 days.
  • The link referenced above will take you to the dashboard, with lots of useful information about your webhooks. If you want to view the delivery logs for a particular topic, click anywhere on the row in the “Webhook topics” table for that topic.

Here’s a great place to get started with Shopify’s webhooks: https://shopify.dev/apps/webhooks.

In Conclusion

Shopify gives app developers several options for staying informed about the current state of a shop’s catalog, supporting both polling and listening for updates. Using both of these approaches helps Skafos ensure that the information we use in our models and display to customers is up to date, allowing us to make the best possible recommendations to each e-shopper.

Try Skafos’s Product Recommendations app now for yourself!


________

Hero photo by drmakete lab on Unsplash