Building a Static Site Generator

The Stratify Logo I designed in 1 sitting, cool isn't it?

I’m a fan of Next.js and Gatsby, both great options for building static sites and dynamic server-rendered sites as well. So I decided I’ll give my own site generator a try.

Let’s call it Stratify, for some reason. We’ll build it as a command-line interface. You can check it out on GitHub or even install it on your own system using npm.

Tech Stack

I’m going to use plain Node.js with plain JavaScript with a few dependencies, we’ll be extensively using the inbuilt fs (file-system) and path packages. We’ll try to keep the dependencies to the bare minimum.

Libraries/Node.js Packages that I’ll be using additionally:

Marked - For conversion from markdown to HTML
Live Server - For running a live server when the user would run the stratify dev command.
fs-extra - For additional utilities that we will need on top of the existing functionality provided by the Node.js fs module.
serve - For serving the build output of the website on a server, when the user runs the stratify start command after building the project.

Adding Commands For Users

So Node.js supports adding binary executables to your package, we want our package to function the way command-line applications do. I.E: Users shouldn’t require to add npm run or node ... before our commands. For that we have the bin part of our package.json file handy. Essentially, the bin section tells the system which commands it has to precompile and which files it should invoke in case a user executes those commands.

You can also use the bin command with the recent runners like npx which don’t even require you to have the package installed in the first place.

{
    ...
    "bin": {
        "create-stratify-project": "./create-stratify-project/index.js",
        "stratify": "./scripts/index.js"
    }
}

On running npm i -g in our directory, we get direct access to the create-stratify-project and stratify commands (We’ll discuss this in the upcoming sections) from our command lines. Users can install our package globally and use the same.

Helping the users setup a boilerplate

As mentioned above, we created a create-stratify-project command, this will help our users setup a boilerplate the way they can with create-react-app and create-next-app.

create-stratify-project <directory-name> [?App Name]

We’ll create a simple file, that contains all pre-packaged files required for the user to get started, linked to the create-stratify-project binary command.

The file can be found here.

Templates

Just Markdown is not appealing on the web, the document generated needs additional info as well. For example, meta tags to tell the browser whether the page is responsive or not, a different title for each page, stylesheets to modify the content that has been generated on the page, in those cases, the power of plain HTML can come in handy.

We can add support for Templates, where a boilerplate is already setup for the user, we simply take the markdown content compiled to HTML and inject it into the appropriate position in the template.

If the user has a page named post-1.md, they could create a templates/post-1.html file and it will be picked up, if a matching template name is not found, the builder defaults to the templates/index.html file. If no templates are found, the html output generated is a simple conversion of the markdown to HTML.

An example of a template would be:

<html>
	<head>
		<title>\{\{ title \}\}</title>
	</head>
	<body>
		\{\{ content \}\}
	</body>
</html>

We inject the conversion output from the markdown file in place of \{\{ content \}\} and the title of the page in place of \{\{ title \}\}. The user can simply add any additional meta tags, scripts, stylesheets they need in this file and they will be included in the build output.

Static Files

Static Files are important for a website, because they don’t necessarily contain code, but are rather files that don’t change very often and hence can be served without a lot of overhead.

We’ll be supporting static files using the public folder like a lot of existing frameworks like Next.js do.

The simplest approach is to move the files from the public folder during build/start time to the same folder where the markdown compiled output is present. That way, a snippet like <img src="/image.jpeg" /> would work properly along with all other snippets requiring static assets.

Another approach is of course, using an express server and setting it to serve all static files from the public folder, this is an approach used by frameworks like Next.js and Create-React-App.

Slight Note for upcoming sections: process.cwd() is extensively used in the code, since this is a command line application, we use the value returned by process.cwd() (Which is the current directory from which the user is executing the program). Similary we will use process.argv which is a way for us to receive command line arguments into our program.

Building a Page

The method to build a page is simple, we’ll create a function that takes the page file’s name and directory as the first argument and the folder to put the build output in.

const buildPage = async ({ fileName, directory }, buildFolder) => {
	const path = require("path");
	const fs = require("fs");

	const pageName = fileName
		.split(path.resolve(process.cwd(), "./pages"))[1] // Get the full file name for the markdown file
		.split(".md")[0] // Remove the .md extension;
		.replace("\\", "/"); // Remove opposite slashes from the path

	console.log("Building page: ", pageName);
	const markdownContent = fs.readFileSync(fileName, "utf-8");
	const { html: convertedHTML, title = "" } = parseMarkdown(markdownContent);

	if (directory)
		fs.mkdirSync(`${buildFolder}${directory}`, { recursive: true });

	// Check if there is any template present for this page.
	const template = getPageTemplate(pageName);
	if (template)
		fs.writeFileSync(
			`${buildFolder}/${pageName}.html`,
			// Replace the  <h1 id="building-a-real-time-online-development-environment">Building a Real-Time Online Development Environment</h1>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fcreating-an-online-development-environment-like-codesandbox%2Fprimaryimage.jpg?alt=media&amp;token=da419413-39fe-4d13-9c15-3542180d07f7" alt="Photo by olia danilevich: https://www.pexels.com/photo/man-sitting-in-front-of-three-computers-4974915/" /></p>

<p>If you’re a developer who’s been working with the JavaScript ecosystem for a long time, you know real-time online development environments are an indispensable part of the development experience. Better yet, if you’re a web developer who has a “not-so-powerful” device, the time to set up or start a project is a huge pain which is alleviated by platforms like <a href="https://codesandbox.io/">Codesandbox</a> and <a href="https://stackblitz.com/">StackBlitz</a> as they take away the entire pain of having to go through the setup process for your apps locally (Which is often the most time-consuming part of the process of getting started with a project) and also provide you with the flexibility to quickly prototype a project, run what you want to and even share samples and code with other people on the internet.</p>

<p>I’ve been a huge fan of Codesandbox from day 1, it is one of the few products that made me go “WOW!” the first time I tried it. Needless to say, the tinkerer inside me wanted to learn how these systems worked internally, I had a fairly good idea but these services do not expose the workings of their systems (Of course) like an open-source project, it would have been great if they did but you can’t have everything in life.</p>

<p>So this post is my journey of figuring out how to create a system similar to Codesandbox.</p>

<p>In this post, we won’t be diving too deep into the code we’ll be using to implement what we discuss, instead, we would just discuss the way we would implement the features or their flows.</p>

<p>For the code, check out the repository at <a href="https://github.com/deve-sh/tranquil">https://github.com/deve-sh/tranquil</a> (Nice name right?).</p>

<p>Let’s go!</p>

<h3 id="laying-down-the-expectations">Laying Down The Expectations</h3>

<p>We can’t build the entirety of Codesandbox or StackBlitz in one go, of course, if we could, they wouldn’t be so special and everybody would do it. In this post, we would only be focusing on building a simple RCE environment that mimics the basic functionality provided to us by Codesandbox, which includes: The ability to create a project from a template, view a list of files, edit them and see the output in real-time.</p>

<p>Let’s lay down some basic functionalities and some technical grounds we would expect from the system:</p>

<ul>
  <li>The user can create a new project, from a template, initially, we can simply have this be a React Project created using CRA or a Next.js project.</li>
  <li>Everything that is not .gitignored is stored in a database as a file entry, with content and other information like:</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
  "id": "&lt;uuid&gt;",
  "name": "&lt;folder1&gt;/&lt;folder2&gt;/&lt;name&gt;.&lt;extension&gt;" // The folders will just be rendered in the UI,
  "content": "..."  // This can be stored separately as well of course
}
</code></pre></div></div>

<ul>
  <li>The user will see a list of files on the left of the screen, an editor at the centre, and a preview screen on the right.</li>
  <li>The code will be connected and running on a remote server, exposing a port on which the client app can show via an iframe.</li>
  <li>Every time a piece of code changes, the primary server makes a write to the database, and sends a signal to a socket on the remote code server with the updated file content, the remote code server writes it to its file system and the process running the app will refresh the application if required using HMR built into the application framework like Next.js and CRA.</li>
</ul>

<h3 id="a-crux-of-how-the-system-works">A Crux of how the system works</h3>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fcreating-an-online-development-environment-like-codesandbox%2Fsecondaryimages%2FReal%20Time%20Online%20Code%20Editor%20Flow1659018680165.png?alt=media&amp;token=237e6912-7779-4793-9f7a-0a9cf9f04771" alt="Real Time Online Code Editor Flow.png" /></p>

<h3 id="what-we-would-need">What we would need</h3>

<ul>
  <li>For the front end, we’d be using React and Vite</li>
  <li>For styling, we would use Tailwind since it makes everything very simple</li>
  <li>Node.js for the main backend</li>
  <li>Socket.io and its client library will be heavily utilized for real-time two-way communication between the client, server and the remote instance running the code.</li>
  <li>A database like MongoDB for file content and metadata storage + retrieval. Could be an SQL Database as well.</li>
  <li>An <a href="https://aws.amazon.com/ec2/">EC2</a> Instance to be spun up on project initialization to run the app.</li>
  <li>An app runner script that will be downloaded on our remote environment EC2 instances which will start the app, set up listeners for logs and crashes and broadcast them via the socket to the backend and through to the client.</li>
  <li><a href="https://codemirror.net/">CodeMirror</a> for the code editor on the front end with key binding support.</li>
  <li><a href="https://www.npmjs.com/package/node-ssh">node-ssh</a> for SSHing into our instance to copy project files into our EC2 instances.</li>
  <li><a href="https://www.npmjs.com/package/wait-on">wait-on</a> to run inside the EC2 instance to wait for the app to come live on the exposed port and notify the backend and client of it.</li>
  <li>A tunnelling software like <a href="https://ngrok.io/">ngrok</a>/<a href="https://localtunnel.me/">localtunnel</a> to expose an HTTPS endpoint for a short-lived dev session from the EC2 instance.</li>
</ul>

<p>Now I know what a lot of you reading this might be thinking, “Why EC2? Why not Docker?”</p>

<p>Well I could use a Docker container, given the machine/remote instance to run the system is just one part of the stack, we could always swap the EC2 Instance for a Docker container.</p>

<p>I chose EC2 simply because of the native API AWS has to create an EC2 instance, but remote code execution services do use Docker containers to quickly spin up instances and execute code and limiting its scope and any vulnerabilities to just the Virtual Machine the container is running on, nothing more.</p>

<h3 id="into-the-technicals">Into The Technicals</h3>

<ul>
  <li>Our React Application rendering the front-end would display a list of projects to the user, fetched from the main server.</li>
  <li>The user selects a project they want to work on by clicking on it.</li>
  <li>At this point, two things happen:
    <ul>
      <li>The front end fetches the list of files and then calculates the id of the last file that was edited on the project.</li>
      <li>The front end requests an endpoint (<code class="language-plaintext highlighter-rouge">/initialize</code>) that spins up a virtual environment server and sends its URL back to the front end.</li>
    </ul>
  </li>
  <li>Using the data the front end received about the project, it renders a view of the files in the project and makes an API Call to get the file’s contents that the user wants to make edits to. In the beginning, it would be the file that was last updated in the project.</li>
  <li>Every time the user makes a change to a file (<a href="https://www.freecodecamp.org/news/javascript-debounce-example/">Debounced</a>, or trigger-based using Ctrl + S, of course), the front-end makes a POST call to the main backend server to store an updated version of the file.</li>
  <li>Once the update is confirmed by the database, the backend makes a file change ping to the app runner script on the EC2 Instance with the updated file contents.</li>
  <li>The RCE server updates its file system and using the process running on its end, re-renders the output. It initiates a ping-back to the front end signalling to it that the IFrame responsible for rendering the application output should re-render using HMR built into CRA, Next.js and other frameworks.</li>
  <li>The RCE server script also listens for crashes, stdout and stderr to send them to the backend server to be forwarded in turn to the client.</li>
</ul>

<h3 id="file-structure-representation-for-projects">File Structure Representation for projects</h3>

<p>We’ll use a simple flat file structure in our backend. For simplicity, we will not store directories individually and instead rely on the structure that AWS S3 uses where a directory is simply a prefix for a file name.</p>

<p>Our frontend will receive files in a flat array like the following:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
	</span><span class="p">{</span><span class="w">
		</span><span class="nl">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59d"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"projectId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59c"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"next.config.js"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"createdAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"updatedAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="w">
	</span><span class="p">},</span><span class="w">
	</span><span class="p">{</span><span class="w">
		</span><span class="nl">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59e"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"projectId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59c"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"package.json"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"createdAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"updatedAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="w">
	</span><span class="p">},</span><span class="w">
	</span><span class="p">{</span><span class="w">
		</span><span class="nl">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59f"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"projectId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"63a7f7ec34daa9b3013cd59c"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"pages/api/hello.js"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"createdAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="p">,</span><span class="w">
		</span><span class="nl">"updatedAt"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2022-12-25T07:12:44.192Z"</span><span class="w">
	</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>

<p>It will process this array on its end to create a nested file structure from it.</p>

<h3 id="background-how-hmr-or-live-reloads-work-for-apps">Background: How <a href="https://webpack.js.org/concepts/hot-module-replacement/">HMR</a> or Live Reloads work for apps</h3>

<p>For the types of apps we would be supporting (Mainly React with <a href="https://create-react-app.dev/">CRA</a> and <a href="https://nextjs.org/">Next.js</a>), HMR comes built-in with the framework using a WebSocket connection established between the app running in the browser and the development server.</p>

<p>The server and the port that runs the CRA app also run the Webpack HMR Server.</p>

<p>In the case of React with CRA, simply creating a connection as a client to <code class="language-plaintext highlighter-rouge">ws://localhost:PORT/sockjs-node</code> will notify us whenever the files change and the server reloads. A similar approach is used by all frameworks like Next.js, Vite etc.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672669061360.png?alt=media&amp;token=b3d27ec8-7df9-4dd8-b3b9-15c17a4ad8e7" alt="image.png" /></p>

<p>The payload for such update events will be of the form:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hash"</span><span class="p">,</span><span class="w"> </span><span class="nl">"data"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>The message event received by the socket client for a file upload and HMR ping:</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668915436.png?alt=media&amp;token=9670db99-1f52-4c3b-b862-28bd931113e0" alt="image.png" /></p>

<blockquote>
  <p>A thing to note is that we don’t need to do any of that manual WebSocket connection setup inside our iframe that we’ll use to display the app to the user, the framework internally takes care of reloading the app on a file change using JavaScript.</p>
</blockquote>

<p>For other apps like Node.js apps, we can utilize Nodemon which provides us with APIs to listen for the process reloads on file changes.</p>

<h3 id="the-entire-app-start-process">The Entire App Start Process</h3>

<p>The app start process is a little intense and lengthy, make sure to click on the image below and read it from start to end.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2FApp%20Start%20Process.svg?alt=media&amp;token=052809be-8634-4516-9364-11f87c49a0f6" alt="The whole app start process" /></p>

<p>There is a lot of information in the above flow, feel free to open the image in a new tab and read through it.</p>

<h3 id="spinning-up-and-terminating-servers-for-running-our-code-on-demand">Spinning up and terminating servers for running our code on demand</h3>

<p>The creation of servers for running our code and then pushing files into it for the project and subsequently readying it to accept further file updates and start our app is the most core part of this project.</p>

<p>To spin up new EC2 Instances we will use the <a href="https://www.npmjs.com/package/aws-sdk">AWS SDK</a> with credentials we can get from our AWS IAM dashboard, we’ll also use the SDK to make other API Calls throughout the life cycle.</p>

<p>We’ll also need to create an <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">AMI</a> with Node.js installed to use as the base for our EC2 instance (Like a Docker image for another image).</p>

<p>We’ll also need a <a href="https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html">security group</a> to expose TCP ports from our instance and of course an <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html">SSH Key</a> to use for logging into the instance and running the app processes.</p>

<p>Before we can SSH into the EC2 instance we need to ensure the instance is healthy and that the health checks have passed for networking otherwise you’ll get a failed response from AWS.</p>

<p>For reference: <a href="https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/ec2-example-creating-an-instance.html">Launching an EC2 Instance with AWS SDK</a>.</p>

<p>We can use the <code class="language-plaintext highlighter-rouge">describeInstances</code> AWS SDK function to check for instance public URL and IP to pass to the front end and store in our database. (<a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeInstances-property">Ref</a>)</p>

<p>We can use the <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#describeInstanceStatus-property">describeInstanceStatus</a> SDK function to check for instance health checks for networking.</p>

<blockquote>
  <p>Note that the health checks and public IPs are not available immediately, so you’ll probably have to ping the AWS API until the data is available, <strong>make sure to wait a few seconds between each call</strong>, it’s important to not hit the rate limit for your APIs in case you accidentally trigger a while loop that keeps hitting the AWS API to get instance status and public URL.</p>
</blockquote>

<p>Once both the above are verified, only then do we proceed to copy our files onto the instance and start our app using node-ssh.</p>

<p>Once the number of socket connections for a project goes to 0, we can use the <a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#terminateInstances-property">terminateInstances</a> SDK function to stop the associated instance.</p>

<h3 id="socket-based-updates-for-project-to-the-front-end-for-logs-and-statuses">Socket-based updates for Project to the front end for logs and statuses</h3>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668324195.png?alt=media&amp;token=a4f0a151-ffed-428d-be8f-1ada0834ef2e" alt="image.png" /></p>

<p>Each client instance for a project will be connected to our backend server. On top of this, the app script running on the remote code server will also be a special type of client.</p>

<p>We would use sockets to send information without polling from the backend to the client, for sending one-time info like file updates we would use simple REST API calls.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668512610.png?alt=media&amp;token=1908e3cc-40f6-4207-a8e6-5bae3b32d925" alt="image.png" /></p>

<p>The backend server will act as the middleman, no connection exists between the client and the app-runner script directly except for the iframe used to show the app, for security purposes.</p>

<p>The client joins the project in a <a href="https://socket.io/docs/v3/rooms/">socket room</a> and receives pings from the backend as updates.</p>

<p>The app-runner script joins a separate room, it sends over logs and app-crash pings to the backend server which verifies the message (Using a secret key added to the message from the app-runner script).</p>

<p>Throughout the entire process of instance spin-up to health checks to project termination, there would be socket-based updates sent to the client to show the user in a terminal window.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668595020.png?alt=media&amp;token=e3087517-19ae-4eae-addd-6b7464b76ad7" alt="A basic look at the first version of socket based terminal output" /></p>

<h3 id="the-code-editor">The Code Editor</h3>

<p>Microsoft has open-sourced its VS Code Editor Interface: <a href="https://github.com/microsoft/monaco-editor">Monaco Editor</a> for the web. This will take care of our requirement of coding and syntax highlighting in the browser, we could even add support for themes to our UI based on the plugins the library supports.</p>

<p>But Monaco is an extremely heavy library with an extremely high level of complexity, and hence, it’s just better to use <a href="https://codemirror.net/">CodeMirror</a> for our simple use case with a controlled editor, where we set the code for the active file received from the backend, allow the user to edit it, and use the <code class="language-plaintext highlighter-rouge">Ctrl + S / Cmd + S</code> key binding to confirm and send the update to the backend and subsequently to the app runner script.</p>

<h3 id="file-updates-for-project-apps-from-the-front-end-to-the-app-runner">File Updates for project apps from the front end to the app runner</h3>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2FHow%20File%20Updates%20over%20the%20socket%20work.svg?alt=media&amp;token=4dbe0d6b-48c4-4009-9886-3bd1b6ad6603" alt="How File Updates over the socket work" /></p>

<h3 id="static-file-uploads-for-projects-in-directories">Static File Uploads for Projects in Directories</h3>

<p>Any remote code execution and development environment are incomplete without the ability to upload files from your system. And we’ll also be supporting this feature.</p>

<p>All files at their core are composed of text. Hence, the file upload will be fairly standard, we would just ask the user to select a file, use browser APIs to get the text, check if the size is less than what we allow and send it as a regular create file operation with initial content as the text retrieved to our server.</p>

<p>The catch is that we only allow type: <code class="language-plaintext highlighter-rouge">text/**</code> and <code class="language-plaintext highlighter-rouge">application/**</code> files for future editing from uploaded files, all other file types are shown as a binary content screen to the user.</p>

<p><strong>Requisites:</strong></p>

<ul>
  <li>An invisible file input.</li>
  <li>An <code class="language-plaintext highlighter-rouge">isReadableContent</code> flag for project files, to be deduced on the front end using the <code class="language-plaintext highlighter-rouge">file.type</code> attribute.
<img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668236225.png?alt=media&amp;token=a9d6b91d-ecc6-4f43-8426-1dd18eb2c740" alt="image.png" /></li>
  <li>Usage of the <code class="language-plaintext highlighter-rouge">Blob.text()</code> method on the front end to read the content of the file and simply invoke the create file endpoint with the content as payload if <code class="language-plaintext highlighter-rouge">file.size</code> is less than 100KB.</li>
  <li>A binary data message to show the user to prevent them from editing or viewing unreadable data for a file.</li>
</ul>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672668252689.png?alt=media&amp;token=69861431-093c-4324-826b-309b59cb8725" alt="image.png" /></p>

<h3 id="environment-variables-for-projects">Environment Variables for Projects</h3>

<p>Apps are incomplete without Environment variables, and I don’t have to explain why.</p>

<p>For environment variables, we’ll store them on the backend and expose them using a Linux script right before the app starts on the EC2 instance via our SSH tunnel from the backend (At the time of instance initialization or server restart).</p>

<p>We’ll provide the user with a place in the project editor to enter a list of environment variables they want to incorporate into their projects. All the communication from the app regarding environment variables will happen over REST.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672669443547.png?alt=media&amp;token=5dff4d7a-b336-49e9-b323-4a98ecb47643" alt="image.png" /></p>

<p>We will obviously encrypt their values using a secret before storage, and once stored, we’ll not send environment variable values to the front end.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672669577910.png?alt=media&amp;token=35527dca-5973-4151-a114-abb1c482e1c1" alt="image.png" /></p>

<p>We can also have protected environment variables like <code class="language-plaintext highlighter-rouge">PORT</code> and <code class="language-plaintext highlighter-rouge">NODE_ENV</code> to prevent manipulation of the dev environment.</p>

<h3 id="project-app-restarts">Project App Restarts</h3>

<p>There would inevitably be the requirement for restarting the app server, it could be because of an environment variable change or an app crash.</p>

<p>In the event an app restart is required, the process will be simple:</p>

<ul>
  <li>We send over a REST API call to the backend notifying it that we need a restart.</li>
  <li>The backend then sends over a socket ping to the app runner server associated with the project.</li>
  <li>The app runner server closes the currently running sub-process for the app and respawns it. Everything remains unchanged, the socket connection is not affected and the logs are streamed from the beginning.</li>
</ul>

<h3 id="dependency-installation">Dependency Installation</h3>

<p>JavaScript apps are incomplete without dependencies. Implementation of dependency installation would be pretty straightforward, we won’t handle installation from UI directly, but rather target the file that’s changed whenever a dependency is installed to a project, which is <code class="language-plaintext highlighter-rouge">package.json</code>.</p>

<p>Any user who wants to install a dependency in the project can simply change the <code class="language-plaintext highlighter-rouge">dependencies</code> object in the <code class="language-plaintext highlighter-rouge">package.json</code> file and save it.</p>

<p>Our front end on detecting a Ctrl + S on <code class="language-plaintext highlighter-rouge">package.json</code> triggers an app server restart (Using the mechanism mentioned in the previous section) with the <code class="language-plaintext highlighter-rouge">npm install</code> command to run before the restart.</p>

<h3 id="https-for-instances-using-tunneling-via-ngrok-or-localtunnel">HTTPS for Instances using Tunneling (Via ngrok or localtunnel)</h3>

<p>HTTPS introduction for our apps on the EC2 instances is a big pain because we have to write scripts to generate certificates, renew them and then apply them over the network for the EC2 instance URL, or do it via Route53 APIs.</p>

<p>All of that is and always has been a huge pain.</p>

<p>This time I took the shortcut of using a tunnelling service called localtunnel, it’s a free option, an alternative to ngrok that allows you to create as many tunnels as needed and supports protocols like Web Sockets out of the box.</p>

<p>The reasons for choosing a tunnel over a static HTTPS connection using SSL certificates were:</p>

<ul>
  <li>Issuing SSL Certificates is difficult.</li>
  <li>The connection to the project would be fairly short-lived, very rarely exceeding over 60 minutes. Hence, it becomes similar to the way we develop apps locally and just use tunnelling software like ngrok to expose it for webhooks usage or testing by other team members.</li>
</ul>

<p><strong>The process looks like this:</strong></p>

<ul>
  <li>In the app runner script, use the <a href="https://www.npmjs.com/package/localtunnel">localtunnel</a> package to create a tunnel to port 3000.</li>
  <li>Send that tunnel URL as a broadcast to the project socket room. Via this broadcast, the front end will update and activate the iframe and all web socket requests for HMR will go through the tunnel URL.</li>
</ul>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672667473941.png?alt=media&amp;token=34535394-d833-48bc-a211-7baf752d12dc" alt="The Tunnel Flow" /></p>

<h3 id="limitation-on-the-number-of-devices-that-can-connect-to-a-project-at-a-time">Limitation on the number of devices that can connect to a project at a time</h3>

<p>To prevent abuse or unnecessary unintentional uses by people who have a habit of having more tabs opened than the days in their lives, we can implement a simple mechanism to limit the connections to a project.</p>

<p>Since we already know the number of connections to a project room, on every new request to join a project room we can check if the number of connections currently is the max. If it is, then we send back a <strong><code class="language-plaintext highlighter-rouge">project-socket-room-rejected</code></strong> status to the client and don’t join them in the room.</p>

<p>The client on receiving that status simply shows the user a message or closes the project editor window entirely.</p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672667729301.png?alt=media&amp;token=7709dfea-853b-43f8-b767-fd30c90cbb5a" alt="The backend socket code listening to check if more than one client is connected and reject any further connections" /></p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672667698591.png?alt=media&amp;token=5db69d46-17fb-4aaa-8e9d-13d904a00e76" alt="Frontend code to receive the project socket room join rejected ping" /></p>

<h3 id="the-rce-in-action">The RCE in action</h3>

<p><a href="https://www.youtube.com/embed/Dqt31d-K6CY">RCE In action</a></p>

<h3 id="webcontainers-an-alternative-to-running-code-in-a-remote-machine-and-instead-running-it-on-the-users-device-in-an-isolated-environment">WebContainers: An Alternative to running code in a remote machine and instead running it on the user’s device in an isolated environment</h3>

<p>After I was done building a big chunk of this project, <a href="https://www.linkedin.com/in/rahulsuresh98/">one of my friends</a> shared this breathtaking post with me from StackBlitz:</p>

<p><a href="https://blog.stackblitz.com/posts/introducing-webcontainers/">Introducing WebContainers: Run Node.js natively in your browser</a></p>

<p><img src="https://firebasestorage.googleapis.com/v0/b/devesh-blog-3fbfc.appspot.com/o/postimages%2Fbuilding-a-real-time-dev-environment-like-codesandbox%2Fsecondaryimages%2Fimage1672667971194.png?alt=media&amp;token=c2fe9463-fe24-40c0-8361-64f3aa66cdcd" alt="Image credits: Stackblitz blog post linked above" /></p>

<p>This was an amazing breakthrough and I feel it solves all the problems people come to associate with remote code execution. Do give it a read! Highly recommended.</p>

<p>If I were to build this system again, I would probably try building it using a similar technology or on top of the <a href="https://github.com/stackblitz/webcontainer-core">open source web container core</a> implementation from StackBlitz.</p>

<hr />

<p>And there you have it, folks, we built our own remote code execution system with templates, real-time project spin-up, file updates, HMR and a few other neat features! The result is not perfect, but it’s not supposed to be. 😉</p>

<p>I hope this post was informative enough, hit me up with any suggestions or feedback.</p>
 block with the converted markdown HTML
			template
				.replace("\\{\\{ content \\}\\}", convertedHTML)
				.replace("\\{\\{ title \\}\\}", title)
		);
	else fs.writeFileSync(`${buildFolder}/${pageName}.html`, convertedHTML);
};

module.exports = buildPage;

In the above example, we’re using some functions as abstractions (I.E: A simple helper function to get templates that might be associated with the page, one to parse markdown and so on, these are available in the GitHub repository previously listed). The function is marked as an asynchronous function so multiple pages can be compiled and built at the same time. For that reference, check out this blog post.

Building the entire site

Now since we have started building one page, we can replicate this functionality and build all pages in a similar fashion, but we’ll also move all contents of the public folder to the build folder so the static assets are available to the .html files that are generated.

// scripts/build.js
async function buildPages(
	buildPath = "./build",
	silent = false,
	exitPostBuild = true
) {
	const readPagesDirectory = require("../helpers/readPagesDirectory");

	const markdownFiles = readPagesDirectory();

	const dirExists = require("../helpers/dirExists");
	const fs = require("fs");
	const path = require("path");

	const buildFolder = path.resolve(process.cwd(), buildPath);
	const publicFolder = path.resolve(process.cwd(), "./public");

	if (dirExists(buildFolder)) fs.rmSync(buildFolder, { recursive: true });
	fs.mkdirSync(buildFolder);

	if (markdownFiles.length) {
		const buildPage = require("../helpers/buildPage");
		const pageBuilds = [];
		for (let file of markdownFiles)
			pageBuilds.push(buildPage(file, buildFolder));

		await Promise.all(pageBuilds);

		if (!silent) console.log("Finished Building Pages");
		if (!silent) console.log("Moving static assets to build directory");
		if (dirExists(publicFolder)) {
			// For all static assets
			const copyAllFolderContents = require("../helpers/copyAllFolderContents");
			copyAllFolderContents(publicFolder, buildFolder);
		}
	}

	if (!silent) console.log("Build successful");
	if (exitPostBuild) return process.exit(0); // Done building without any issues
}
module.exports = buildPages;

Dev Server

We have two options to do this:

Create an express app internally that serves the static assets from public folder, and on each request, compiles markdown from the requesting page, converts it to HTML and serves it.
Use an existing utility package like live-server that we feed a .live folder, which contains the build output from the pages directory and the public directory. We listen for changes to any files in the pages directory using fs.watch and build that specific page, put it into .live and let the live server do its job.

For me, the simpler approach was the second one, however, watch out for a beta version of the package in case I decide to try out approach one.

You could also try Webpack’s Hot Module Reload, but I’ve never been a fan of Webpack and its working so I refuse to give it a try.

Check out the file responsible for running the dev server here.

Serving a Build

One we have built the site into html files from the pages directory into the build folder. There’s a very simple package we need to use in order to serve a build: serve. It can be run directly using npx run serve so all we need to do for our use case is execute that command.

// scripts/start.js

// Check if there is a build folder, if yes, use the 'serve' package to serve it's content on a local server.
function start() {
	const path = require("path");
	const dirExists = require("../helpers/dirExists");

	if (dirExists(path.resolve(process.cwd(), "./build"))) {
		const { execSync } = require("child_process");
		execSync("npx serve build", { stdio: "inherit" }); // stdio: inherit means all the input output will be of the command that execSync is running.
	} else
		console.log(
			"There is no build folder. Run npm run build to build your pages."
		);
}

module.exports = start;

Putting it all together

Since we now have all functionality required in order to develop, build and serve a static website generated using Markdown. We can setup the scripts interface, which will actually make stratify dev, stratify build and stratify start work.

scripts/index.js:

#!/usr/bin/env node

const commandType = process.argv[2];

if (!commandType || !["dev", "build", "start"].includes(commandType))
	return console.log("Use stratify dev/build/start for appropriate action.");

if (commandType === "dev") {
	// Start the dev server, with live reloading for changes in the 'pages' directory.
	const dev = require("./dev");
	dev();
} else if (commandType === "build") {
	// Build the pages directory and generate a 'build' folder.
	const build = require("./build");
	build();
} else if (commandType === "start") {
	// Check if there is a build folder, if yes, use the 'serve' package to serve it's content on a local server.
	const start = require("./start");
	start();
}

That’s it, we publish the package, and once we run npm i -g stratify-web, the binary executables for the package will be ready for everyone to use as highlighted here.

Check out the package source code

Check out the package published on npm