Building an “easy” web application

I decided to spend my Fourth of July weekend transforming a small Python utility I wrote a couple of years ago for work. The idea was simple: make the tool more accessible for co-workers who use it by turning it into a web application.

The premise of the tool is extremely simple. It automates the sending of multiple cURL requests to a WordPress site’s xmlrpc.php file with some extra QOL features.

The purpose of automating the sending of multiple requests is that a single cURL request may succeed, but the WordPress mobile app and Jetpack do not make single requests – they make many requests and need to be in constant communication with that file. A single request might succeed, but many can fail due to rate limiting, bandwidth or other hosting limitations. It is impossible to fully diagnose a possible XML-RPC issue without performing a test like this.

The utility works just fine, but I thought converting it to a web app would make it easier for folks to use. My first thought was to simply write a PHP script to make the POST request then return the results to the client. It probably would have taken all of 30 minutes but I thought “nah that’s too much work for a simple app….” So I thought I’ll just port the Python script into a web app using a Python web framework. Django is far too much for something like this, so I started looking at Flask. Even Flask seemed a bit overkill for a one-file project; thankfully there’s Bottle which is a “micro-framework” similar to Flask.

I decided to spin up a Bottle app which is ridiculously easy and had a working version of the script running on a web page in about an hour.

It worked just fine, but I wasn’t sure I liked the idea that the user would have to wait during the request for the script to complete all n number of requests before the results page loaded. I also thought that JavaScript might be a better solution in general: I could just make the requests via JavaScript and append the results to the DOM as they came in. I wouldn’t even need a server component. It would be so simple…

The first thing I did was re-write the request using JavaScript’s Fetch API. This took a little while because I needed to refresh on Async Functions and Promises. I feel like every time I look at JavaScript it’s different 😭. After translating the request to a JavaScript acceptable request:

let mySite = document.getElementById('siteURL');
    
    let myRequest = await fetch(mySite, {
        headers: {
            "Content-Type": "text/xml",
            "User-Agent": "Jetpack by WordPress.com",
        },
        body : '<?xml version="1.0"?><methodCall><methodName>demo.sayHello</methodName><params></params></methodCall>',
        method: "POST",
    });

The next step was to wrap it into an Async function for looping:

const MyAsyncFunc = async () => {
event.preventDefault(); //don't reload the page
}

And then wrap the previous code into a loop and append the results to the DOM.

const MyAsyncFunc = async () => {
event.preventDefault(); //don't reload the page

  for (let i = 0; i < 50; i++){
    
    let mySite = document.getElementById('siteURL');
    
    let myRequest = await fetch(mySite, {
        headers: {
            "Content-Type": "text/xml",
            "User-Agent": "Jetpack by WordPress.com",
        },
        body : "<?xml version=1.0?><methodCall><methodName>demo.sayHello</methodName><params></params></methodCall>",
        method: "POST",
    });

    //console.log("Request #" + i + "HTTP Response: " + myRequest.status + " " + myRequest.statusText);

    document.getElementById("results").innerHTML += `<div>${"Request # " + (i + 1) + " | HTTP Response: " + myRequest.status + " " + myRequest.statusText}</div>`;
    }
}

This worked GREAT except:

The request was coming from a different origin than the server, so it was blocked by CORS which is a security feature imposed by all modern browsers. From the wiki:

Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served.

A web page may freely embed cross-origin images, stylesheets, scripts, iframes, and videos. Certain “cross-domain” requests, notably Ajax requests, are forbidden by default by the same-origin security policy. CORS defines a way in which a browser and server can interact to determine whether it is safe to allow the cross-origin request. It allows for more freedom and functionality than purely same-origin requests, but is more secure than simply allowing all cross-origin requests.

– Cross-origin resource sharing

Shit. I was never going to be able to be in control of the server from which the client request originated, since this tool in essence needs to be able to query any WordPress site. I understand the security implementations, but was kind of frustrated. I could get exactly what I needed in one cURL request, now I had to find a way to work around this. After some discussion on the Handmade Network, I confirmed that the request could be made from node or another tool on the server where CORS doesn’t exist for the request to the target WordPress site, and I would control the CORS between the client and the server.

This quickly became more complicated than it needed to be. I could have just run the information through the back end and load it into a new route but that would be just like the Python solution. The idea was to run the request async and append the results to the DOM.

Of course, the actual request would look something like this:

Except it’s even more complicated because you can’t just pass something to the server like that and get it back asynchronously…. you have to set up an API! So I set up a small REST API to receive a GET request appended to the URL as the URL. Then sent that URL off in a fetch request, then sent the promise back to the client as the response data:

app.get("/:url", (req, res, next) => {
    var recievedURL = req.params.url;
    console.log("Got URL: " + recievedURL);
    fetch(recievedURL, {
        headers: {
            "Content-Type": "text/xml",
            "User-Agent": "Jetpack by WordPress.com",
        },
        body : '<?xml version="1.0"?><methodCall><methodName>demo.sayHello</methodName><params></params></methodCall>',
        method: "POST",
        mode: "cors"
    }).then(data => res.send(data));
    console.log("Sent Data");
});

Yeah, it was messy as shit. It was also getting harder and harder to debug as JavaScript has no timeout functionality I would have had to wrap that request in another promise. It was all just so needlessly complicated and frustrating. I kept going back to my command line and running that single cURL request, getting exactly what I needed, and getting pissed.

Ultimately I decided that the node app + REST API was just more trouble than it was worth. Two packages installed and I had a bloaty node_modules directory with 73 folders and 335 files.

Ultimately I said “screw it” and decided the whole API approach for this one-page app was WAY more trouble than it was worth. I would have had to maintain the client, server, and node packages because you know those things have to be updated every 9 minutes or you have a security vulnerability 🙄.

So, 72 hours later and I was back to where I started, with my simple yet effective one file bottle app. The source is 6kb and can handle what it needs to, the error logging is verbose and it works how I expect it to. The tradeoff? People have to wait 1-30 seconds for the request to complete or timeout, so I added a GIF to help them with the wait:

My friend & co-worker Brooke helped put some simple styling on it (also helped with my workout program design) and I think the final result looks pretty sleek:

I also got to do all of the fun systems stuff like remember how Apache works, update certbot from ACME v1, and play with the virtual hosts file which is always a good time.

I spent some serious time on input sanitation and error catching, so it should be interesting to see how people break it 🙂