Devographics Monorepo

This is the codebase that runs the Devographics surveys, such as State of JS and State of CSS.

Setup

1. Install pnpm

This monorepo uses pnpm, and you should start by installing it.

2. Clone Repositories

A. Install Script

You can clone all repositories in the right place and create blank .env files with the following install script:

curl -o- https://raw.githubusercontent.com/Devographics/Monorepo/main/install.sh | bash

B. Manual Install

Monorepo

• Clone this monorepo locally with git clone https://github.com/Devographics/Monorepo.git

Other repos

The survey apps rely on a lot of metadata. If you need to load or modify this metadata from your local filesystem instead of through our API, you can optionally also clone these other repos:

1. entities: the YAML files containing all metadata for the features, sites, people, libraries, etc. mentioned in the surveys.
2. surveys: contains the YAML files that define survey configs and outlines.
3. locale-en-US: (or any other locale) contains the locale strings used in the survey.

I suggest using the following file structure:

• devographics parent directory
  • devographics/monorepo
    • devographics/monorepo/surveyform
    • devographics/monorepo/results
    • devographics/monorepo/...
  • devographics/entities
  • devographics/surveys
  • devographics/locales
    • devographics/locales/locale-en-US
    • devographics/locales/locale-fr-FR
    • devographics/locales/...

3. Install Dependencies

Whenever you need to run a project in the monorepo directory (such as results, surveyform, etc.), you will need to first run pnpm install from within its subdirectory.

• cd monorepo
• pnpm install

This will install dependencies for all applications of the monorepository.

4. Running a Project

You can run each project with pnpm run dev when inside its subdirectory.

For example, for the surveyform app:

• cd monorepo
• cd surveyform
• pnpm run dev

If this is your first time running a project, you will run into some issues that need to be fixed by configuring your local environment variables (see below).

Remote vs Local API

You can run the results and surveyform project by connecting to our remote production API (https://api.devographics.com/graphql).

Alternatively, if you want to use local files for the surveys, locales, etc. you will need to run a local copy of the API project (monorepo/api) locally as well to load them. Once the API is running, you can then point the other apps to it via the API_URL env variable.

Note that even when the API is running locally, you will still need internet access to connect to the databases and load image assets.

5. Environment Variables Setup

Each app within monorepo needs its own environment variables defined inside a .env files (except for surveyform and surveyadmin, which use a .env.development.local file).

Here are some ways that can help with this setup:

1. You can use the .env.example file in each project subdirectory (such as this one for the surveyform project) as a starting point by pasting its contents into your own .env or .env.development.local (for Next.js projects such as surveyform) file.
2. You will need credentials to connect to our MongoDB and Redis staging database. You can ask me (Sacha) on Discord and I will provide them.
3. When running the app with pnpm run dev you will get error messages indicating any remaining missing environment variables.

You can also refer to variables.yml directly and look for variables corresponding to the current app (e.g. results).

Apps

The following apps are all contained within the monorepo.

📡 API

Node.js TypeScript app.

What It Does

• Make the outlines of each survey available to the survey form app.
• Connect to the database and generate the data for the results app's charts.
• Provide the internationalisation strings for each locale.

Code

• https://github.com/Devographics/Monorepo/tree/main/api

Hosted On

• https://render.com

🔍 GraphiQL

GraphQL IDE

What It Does

• Make it easier to test and query the API.

Code

• https://github.com/Devographics/Monorepo/tree/main/graphiql

Hosted On

• https://netlify.com

Domain

• https://api.devographics.com

✍️ Surveyform

Next.js TypeScript app.

What It Does

• Let respondents take the survey.

Code

• https://github.com/Devographics/Monorepo/tree/main/surveyform

Hosted On

• https://vercel.com

Domain

• https://survey.devographics.com

📊 Results

Gatsby TypeScript app.

What It Does

• Display the survey results.

Code

• https://github.com/Devographics/Monorepo/tree/main/results

Hosted On

• https://netlify.com

Domains

• https://2022.stateofjs.com
• https://2023.stateofcss.com
• etc.

🔒 Surveyadmin

Admin app.

What It Does

• Provide a dashboard to manage all surveys.
• Handle data processing and normalization.

Hosted On

• Only running locally for now.

---

Databases

🗂️ Main Database

MongoDB database.

What It Does

• Store the raw data entered by respondents.
• Store the "normalized" data once it's been processed.

Hosted On

• https://www.mongodb.com/atlas/database

Local dev

You can plug to a staging database or run Mongo locally via Docker.

See "docker-compose.yml" and "Justfile" for our local setup. When running locally, data are stored in a ".mongo" folder in the monorepo folder. You can delete this folder to reset the local database.

💾 Cache Database

Redis database.

What It Does

• Cache the results of queries made to the API app.

Hosted On

• https://upstash.com/

Local dev

You can plug to a staging database or run Redis locally.

Since we use Upstash, which rely on HTTP requests instead of direct Redis connection, we also setup an HTTP proxy.

See "docker-compose.yml" and "Justfile" for our local setup.

To reset the local Redis instance, the best approach is to remove the Redis container.

---

Assets

🌐 Locales

Locale strings

What It Does

• Store locale strings for various languages as YAML files.

Repos

• https://github.com/Devographics/locale-en-US
• https://github.com/Devographics/locale-es-ES
• https://github.com/Devographics/locale-de-DE
• etc.

📖 Survey Config

Outline and config files for each survey.

What It Does

• Store outline and config files for each survey as YAML files.

Repo

• https://github.com/Devographics/surveys/

Domains

• https://assets.devographics.com

🖼️ Static Assets

Static image files.

What It Does

• Store static images such as logos, social media preview images, etc.

Repo

• https://github.com/Devographics/images

Hosted On

• https://netlify.com/

Domains

• https://assets.devographics.com

Contribute

Emojis to distinguish commits within the monorepo:

• 🅿️ :parking: for the whole monorepo ("P" for "Pnpm")
• ⚙ :gear: for the shared folder
• 🔍 :mag: for graphiql
• 📡 :satellite:for the api
• 📊 :bar_chart:for the results
• 🏠 :house:for the homepage
• ✍ ️:writing_hand: for the surveyform
• 🔒 :lock: for the surveyadmin
• 🌐 :globe_with_meridians: for the locales
• 📖 :book: for the surveys
• 🙎 :person_pouting: for the entities
• 🖼️ :frame_photo: for the images

---

Env Variables

See "shared/helpers/variables.yml" for a more up to date list.

API

Variable	Description	Used By
API_URL	URL of the API	All

MongoDB

Variable	Description	Used By
MONGO_URI	URI of the Mongo database	All
MONGO_PRIVATE_DB	Name of the database where private data is stored	Surveyform
MONGO_PUBLIC_DB	Name of the database where public data is stored	API

Redis

Variable	Description	Used By
REDIS_URL	URL of the Redis database	Surveyform
REDIS_TOKEN	Redis token (needed for Upstash)	Surveyform

GitHub

Variable	Description	Used By
GITHUB_TOKEN	GitHub access token	Results
GITHUB_PATH_SURVEYS	Path to surveys dir on GitHub (e.g. org/repo/(subdir))	Results
GITHUB_PATH_LOCALES	Path to locales dir on GitHub (e.g. org/(repo)/(subdir))	Results, API

• For GITHUB_PATH_SURVEYS, the subdir segment can be omitted. It will then be assumed that the surveys data is at the root of the repo.
• For GITHUB_PATH_LOCALES, both the repo and the subdir segments can be omitted. It will then be assumed that the locales each have their own separate repo.

Email

Variable	Description	Used By
EMAIL_OCTOPUS_APIKEY	EmailOctopus API key	Surveyform
DEFAULT_MAIL_FROM	Default "from" email (e.g. [email protected])	Surveyform
SMTP_HOST	SMTP host (e.g. email-smtp.us-east-1.amazonaws.com)	Surveyform
SMTP_PORT	SMTP port (e.g. 465)	Surveyform
SMTP_SECURE	Set to "1"	Surveyform
SMTP_USER	SMTP username	Surveyform
SMTP_PASS	SMTP password	Surveyform

EMAIL_OCTOPUS_APIKEY

Other Config

Variable	Description	Used By
ENCRYPTION_KEY	Encryption key to hash emails	Surveyform
SECRET_KEY	Secret key used to verify external webhook requests	API
ASSETS_URL	URL for static assets (e.g. https://assets.devographics.com/)	All

Local Dev

Variable	Description	Used By
SURVEYS_DIR	Local directory from which to load survey outlines	API
LOCALES_DIR	Local directory from which to load locale files	API
ENTITIES_DIR	Local directory from which to load entities files	API
ENABLE_CACHE	Set to false to always load data from the API	All
PORT	Which port to run the app on	All
LOGS_DIRECTORY	Absolute path to logs dir (e.g. /Users/sacha/monorepo/surveyform/.logs)	All

• If SURVEYS_DIR is defined, surveys data will be loaded locally instead of from GitHub.
• If LOCALES_DIR is defined, locales data will be loaded locally instead of from GitHub.
• If ENTITIES_DIR is defined, entities data will be loaded locally instead of from GitHub.

Other 3rd party services

• Plausible for analytics

Glossary

A glossary of terms used internally throughout the code and apps.

Answer

A single answer to a survey question. Note that some questions accept multiple answers.

📡 API (app)

The Node.js app that serves the GraphQL API which in turn gives all other apps (as well as any external client) access to the data.

_cardinalities

A special auto-generated API field that returns how many respondents use 1 item in the current section, 2 items, 3 items, etc.

Code

In the context of processing (or “coding”) freeform data, is synonymous with Token.

Combined Data

A dataset containing both freeform and fixed responses.

Comments

Optional comments left by respondents in relation with a specific question.

Config File

A config.yml file used to configure a given survey or survey edition.

Dynamic Options

For freeform questions that do not have a pre-established list of options, a list of options can be generated based on the dataset dynamically. This is useful for filtering.

Edition

A specific edition (i.e. year) of a survey.

👨 Entities Repo

The repo containing all entities definitions.

Entity

Any item, person, concept, etc. that can potentially be referenced in the surveys. Defined in the entities repo.

When used in opposition to Token, indicates a “real” item such as a feature, library, person, tc. with a homepage, social media links, or other associated metadata.

Explorer

The GraphiQL Explorer can be used to navigate the GraphQL API.

Freeform Data

Data resulting from a freeform text area and not a fixed list of questions.

GraphiQL

The IDE that lets you query the API.

GraphQL

The protocol used to define and query the API.

🏠 Homepage (app)

The app used to generate an individual survey's homepage (such as https://stateofjs.com).

_items

A special auto-generated API field that returns all items belonging to the current section.

🌐 Locales Repos

A collection of repos (such as locale-en-US) containing localized content for the surveys.

_metadata

A special auto-generated API field that returns metadata about the current item.

Normalized Data

Data that has gone through the normalization process, which includes:

• Removing empty responses.
• Assigning entities/tokens to freeform answers.
• Copying/renaming fields.
• etc.

Normalized Data Collection

The MongoDB collection (a.k.a. Database Table) that stores normalized data, as created by the Surveyadmin app.

This collection or its contents can be made public, either through the GraphQL API or by sharing CSV or JSON files containing a survey edition's entire dataset.

Options

The list of fixed options available to choose from to answer a given question.

See also: Dynamic Options.

Patterns

Regex-defined string matching patterns (in a question's patterns field) used to match freeform data to entities.

Rules of patterns:

• An entity's id is automatically used to match strings.
• Any _ in the id can also match - or spaces.
• Matching is case-insensitive.
• Patterns automatically match their plural form (animation will also match animations).
• Patterns can also use the following modifiers:
  • [p] (partial): Match partial word fragments.
  • [l] (list): Comma-separated list of items to match in any order.
  • [e] (entire): Match entire answer exactly.
  • [w] (whole): Match whole words (default).

Predefined Data

Data resulting from a fixed list of options (e.g. multiple choice questions, dropdown questions, etc. ).

Prenormalized Data

Data resulting from a practically unlimited, yet still predefined list of options (such as a list of all Best of JS projects, or all GitHub repos).

Query Builder

A client-side query-building UI provided in the Results app that lets end users dynamically build and run GraphQL queries to generate custom charts.

Question

A question asked in a survey.

Question Response

A response to a specific question by a specific respondent. Note that some responses can contain more than one answer.

Questions Outline (or: Sitemap)

A YAML file definining the shape of a survey's questions. Stored in the surveys repo.

Raw Answers (or: Raw Data)

The raw list of answers for any given question.

Response (or: Survey Response)

The entire response document containing all of a single respondent's data for a given survey.

📊 Results (app)

The app used to compile a dataset into a static, public-facing results site.

Results Outline

A YAML file defining the shape of a survey's results site. Currently stored in the monorepo repo but should eventually migrate to the surveys repo.

Section

A survey section.

⚙️ Shared (monorepo sub-directory)

A sub-directory of the monorepo containing code shared by other sub-apps (and imported as local packages via PNPM).

Survey

A specific kind of survey (State of JS, State of CSS, etc.).

🔒 Surveyadmin (app)

The app used internally to normalize surveys datasets and run scripts.

✍️ Surveyform (app)

The survey form app used to collect survey responses.

📖 Surveys Repo

The repo containing all survey definitions and associated data.

Tag

An optional tag that can be assigned to an entity to categorize it. This can then be referenced by a question to select a subset of entities.

Rules of tags:

• Entities defined in foo.yml automatically get assigned the tag foo.
• Entities also automatically get assigned the names of all parent directories as tags.
• Other tags can be explitly defined in an entity's tags field.
• Questions can reference tags in their matchTags field.
• Questions automatically reference any tag corresponding to their own id field.

Template

In the context of question definitions (either in the questions outline or results outline), a set of predefined parameters that a question can extend to keep code DRYer.

Token

A special sub-type of entities that correspond to concepts (such as “hard learning curve”) and not “real” items.

Unnormalized Data

Data that has not gone through the normalization process yet.

Unnormalized Data Collection

The MongoDB collection (a.k.a. Database Table) that stores unnormalized data, as provided by the Surveyform app.

This collection or its contents should never be made public.