Happy Internal Customers through Diffgram’s Development System

  • Do you get a feeling the choice is more important then it first seems?
  • Why is there so much attention around this area?
  • Is it more then just another tool?
Pictured, New York, Sept 2022

Diffgram Development System

The story to surface here is that Diffgram is not just a routine tool purchase. Diffgram is a Development System you use to create and maintain your own platform.

Why you need a way to develop it

It’s tempting to think of Training Data as just something to leave up to the data science team, but the reality is this is a choice that effects your entire company.

Why does Training Data not feel like just another IT tool?

Consider that most of the following is probably true for the your business:

  • Training Data touches your most sensitive raw data
  • Training Data is a key part of your R&D, or even core product, efforts
  • Training Data touches nearly everyone in your IT department, from engineering through infra and data science.
  • Training Data, has changing business requirements and a growing volume of users across over multiple business units.
  • Training Data, as a growth area, has an increased level of executive awareness

That nagging sense of something being “off”

As you have been exploring options you have likely seen slick websites, videos, sales presentations. Yet you may feel a nagging feeling of “if it’s really so good, why isn’t it flying off the shelves?”. You may be also wondering why “all-in-one” solution for training, modeling, etc. also aren’t taking off more?

Something doesn’t add up…

Internal Customer to Area of Training Data Map

This table maps an Internal customer group and leader to the areas training data effects.

Map of Internal customer group to the areas training data effects.

Many People, a Major Choice

Training Data is software that touches:

  • tens of people in an administrative way
  • tens and hundreds of end users today
  • In the future, a noticeable percent of your companies activity will in some way involve ML programs, which will in some way involve training data.
Training Data competency effects cascade through your company

The Software is Not There Yet — Out of the Box

What most everyone in the Training Data industry knows, but rarely says, is that there is no such thing as a complete, out of the box, solution yet.

Training Data software is still being Developed.

Unknown Unknowns

As a growing area the requirements are expected to change over time. How can you be a strong position to adapt to those requirements? Just trust the vendor? You have been around the block so I’ll let that one speak for itself…

Summary need for a Development System

So to summarize this

  1. Training Data is more then a routine IT purchase
  2. Training Data effects the majority of your internal customers
  3. Training Data software is not there yet, so a Development System is needed.

Development System

Here’s how it works.

The illustration uses scaffolding as a metaphor for AI’s quest in unearthing the underlying logic and structure of complex organic matter. Artist: Khyati Trehan.

Blueprint

The Development System has these parts:

  1. Baseline Platform, with End User level customization
  2. Develop Within Diffgram Frameworks
  3. Develop Novel Software using Diffgram Components
  4. Develop Novel Software on top of Diffgram
  5. Prioritized feature requests (Diffgram Developing)

Baseline Platform

Diffgram offers a standard set of features similar to others. Diffgram has one of the most customizable and configurable End User accessible platforms. What is described in this article is “extra”. I won’t spend any more time on the Baseline Platform or End User Configurations in this article, those are both well described elsewhere.

Diffgram Frameworks

In theory everything is an integration. In reality, as you know, the degree of pre-existing work matters greatly.

  1. Scripts for Frontend
  2. Workflows for Backend
Workflow example apps to install
  1. Install Open source ML Programs (Ex. Hugging face, Deepchecks)
  2. Big tech cloud providers (ex. Google Vertex AI)
  3. Your own in house methods (Example, Interactive engagement by calling API from script).

Develop Novel IP using Diffgram components

The Diffgram Frameworks provide standard ways to install important sub processes. What happens when you want to start working with the UI, or other parts, and developing directly in novel ways?

Storybook, Example of a Dataset Selector with Mock Request.
Construct Scene from the Matrix

Directly Develop Novel Software from Diffgram

Optionally, we offer the most flexible possible, which is engineering directly on top of Diffgram source code.

Vendor Developing

We have been continuing to refine our process for working directly with your team on Prioritized Feature requests. Please note this is not blanket consulting, rather we take your needs, and abstract them into more general forms. This allows our development to be grounded in business need, and lowers the development cost for you. I won’t belabor that process here, but it is much more budget friendly then regular consulting add on costs.

Happy Internal Customers

So how does this all come together to keep your internal customs happy? What are the benefits?

Happy CFO

Investing in your own Diffgram platform keeps the CFO happy in a few ways:

  1. Keeps custom development costs under your control
  2. Keeps predicted future costs more under your control

Happy CISO

Security has many layers. Trusting a SaaS solution just doesn’t make sense for this type of sensitive data and company reaching effects.

Happy CEO

  1. Better Control
  2. Better Aligned Economics
  1. Your own existing staff can be engaged for better control
  2. You have a great degree of control, meaning you can trust high value workloads to it
  3. The future usage is less directly dependent on a 3rd party vendor
  1. Some vendors are being acquired by an annotation firms (two in the past few years.)
  2. Some vendors will likely be acquired by defense contractors

Happy CTO

Own Installation by Default

Summary

The Diffgram Development System has emerged out of years of work. It’s a bow tie, wrapping many concepts that solve individual business unit problems. We plan to expand the Diffgram Development system over time.

  • Diffgram is a Development System you use to create and maintain your own platform.
  • Training Data effects the majority of your internal customers. The Software is not there yet out of the box by any vendor and some of the needs are still unknown. Meaning that a Development System, a strategy, is needed.
  • The Development System gives your team higher levels of Development control and other benefits then customization or API integration alone.
  • There are many tangible and obvious benefits to your Internal customers. From managing development costs, and future costs for the CFO, security customization and re-use of existing controls for the CISO, unlocking new product cases for Engineering and R&D, and overall providing a better control and direction to the CEO’s office.

Getting Started Adopting the Diffgram Development System

Here’s the best news. Getting started is as easy as choosing to adopt the mindset of the Diffgram Development System. The technical implementations and team training etc. can be adopted incrementally over time as needed.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store