How does your company use Node.js? (I'll show you mine if you show me yours)

Hi, long time lurker here. I personally find it difficult to get a read on patterns businesses use for node.js. Obviously it's easy to find patterns that are common throughout open source projects, but these are mostly frameworks rather than large applications intended to be hosted and run somewhereSo, let's compare notes! I'm going to give a short summary of the technical domain of the company I work for, how many employees and then an overview of patterns we use within the code and if I'm lucky some of you will too :)What we doThe company I work for is basically a big ETL pipeline, pulling in data from various sources. Data is filtered, transformed and finally aggregated into a search engineWe're a startup and so only have 3 developers including me. We've been working on our technology for 3 years. In our first 2 years, we played about with Python, Node.js and Clojure. About a year ago we decided to settle on Node.js. The three of us are relatively new to Node.js but have picked up the basicsHow we doProject structureThese files are just made up of course but show the general structure- src/ - todos/ # some of our projects are monoliths and some are (micro?)services. Monolith projects have separate components in different folders and code is kept separate to reduce monolith complexity - route/ - postTodoRoute.js # Express route - getTodosRoute.js - worker/ - removeObsoleteTodosWorker.js # job that runs on a schedule. Generally just setInterval. Now that node.js has workers, this is a bad name 🐶 - service/ # all code that talks to external dependencies. Useful to keep them separate by convention so it's clear what needs to be stubbed in integration tests - postgres/ - getTodosService.js - datadog/ - auth0/ - transform/ - pgTodoTransform.js # transform a Todo from postgres representation to in-memory - models/ - todoModel.js # though our code is not object-oriented, we still represent common data structures in `class`es as they work well with JSDoc and editors tend to be able to autocomplete fields - dependencies.js # an object that contains all side-effect dependencies. This can be stubbed in integration tests and is passed to routes as req.dependencies and passed to workers as a function arg - utils/ # other stuff - router.js # express router for routes exposed by this component - index.js # set-up workers and any other initialisation logic for this component - someOtherComponent/ - test/ - todos/ - integration/ - unit/ Coding styleOur code is sort of procedural, sort of functional. By this I mean that we very rarely mutate state and that code that has external side-effects is confined to services/ folderI'll give some contrived examples:src/todos/route/getTodosRoute.js:async function getTodosRoute(req, res, next) { try { logger.info('todos/routes/getTodosRoute() - starting..', { 'req.query': req.query }); const result = Joi.validate(req.query, GetTodosRequestQuerySchema); if (result.error !== null) { throw result.error; } const pgTodos = await req.dependencies.getTodosService(req.query); const transformedTodos = pgTodos.map(pgTodoTransform); const response = { todos: transformedTodos, }; logger.info('todos/routes/getTodosRoute() - finished'); return res.json(response); } catch (error) { logger.error('todos/routes/getTodosRoute() - error', { error }); return next(error); } } module.exports = getTodosRoute; src/todos/worker/removeObsoleteTodosWorker.js:function create(dependencies, frequencyInSeconds=60) { // again, a worker is not a Node.js Worker. We just foolishly named these workers const worker = new RemoveObsoleteTodosWorker({ depencencies, eventEmitter: new EventEmitter(), intervalId: null, state: {}, }); worker.eventEmitter.on('start', () => { worker.intervalId = setInterval(async () => { try { logger.info('todos/workers/removeObsoleteTodosWorker/create() - starting iteration..'); worker.state = await onIteration(worker); worker.eventEmitter.emit('iterated'); // used by integration tests to inform when worker has iterated logger.info('todos/workers/removeObsoleteTodosWorker/create() - finished iteration'); } catch (error) { logger.error('todos/workers/removeObsoleteTodosWorker/create() - error during iteration', { error }); worker.eventEmitter.emit('error', error); } }, frequencyInSeconds * 1000); }); return worker; } function start(worker) { worker.eventEmitter.emit('start'); } module.exports = { create, start, }; A word on componentsSome of our components are separate microservices and some are simply modular bits of code within a monolith. Right now these only ever talk to each other via REST interface or by setting state in an integration database. We're looking into more robust and asynchronous communication soon like RabbitMQ or SQSWould be VERY interested to hear how other companies deal with inter-component communication :)Commonly used frameworks / librariesexpressmocha/chai/sinoneslintlodashjoi (https://ift.tt/2PXsXuJ Schema validationmoment-timezone (https://ift.tt/2O7S6Wi Datetime / timezone stuffdotenv (https://ift.tt/2PXsYyN Loads env vars for local development. We store the .env files in a password manager so they never touch version controlnose-sql-template-strings (https://ift.tt/2O1cbxC We just directly write SQL rather than using an ORM, as we have some bizarrely esoteric queries. This library makes sure its escaped and sanitiseddb-migrate (https://ift.tt/2PXtnRP Database migration scripts. We use --sql-file so that migrations are directly written in SQLwinston (https://ift.tt/2O40nus Logging. We log as a JSON so that any log service can parse and search structured dataTl;drWasn't expecting to write that much but hope it helps someone else also wondering how other companies do itVery excited to hear people's thoughts and even moreso how they do it :D

Submitted September 22, 2018 at 08:43PM by chaptor