Handling large JSON payloads and extending the Request class
September 05, 2024
In one of the previous articles, I've discussed how you could accept and validate Base64 files within Laravel: Send, validate, and store Base64 files with Laravel. Now imagine a situation where you would like to allow your users to upload large (let's say up to 50 MB) images through your API in a Base64 format. In order to achieve that, they would first have to convert their 50 MB image to Base64 and include that generated string inside a JSON payload, targeting your designated route with a POST request. Simple enough, right? Not quite. Let's see what problems we can encounter and how to deal with them. I just want to note that I'll be using Laravel 11, and that some lines of code might be different if you are using an older version of the framework, but the general principle should be the same in most cases.
Note: before we start, this article uses an approach that increases the memory limit when specific routes are targeted, and it's likely suited for apps that have a few routes that need to process larger files with a maximum of 50 MB (roughly speaking) or less. Please bear in mind that this is just one of the strategies you might want to implement. Most likely, you wouldn't use this approach (or at least not on its own) when processing files that might be a lot larger than 50 MB. You don't want to increase memory indefinitely. In that case, you might consider using alternative approaches like streaming and chunking the Base64 string, or, if possible, implementing chunking on the client side. You could also implement more than one of these strategies, depending on your requirements.
Dealing with PHP and NGINX limits
As you may know, by default PHP won't allow you to process that kind of large payloads and files without tweaking a few settings. So, you'll most likely end up increasing these two values in your php.ini file:
upload_max_filesize = 50M
post_max_size = 70M
and if you are using NGINX, you'll probably need to increase client_max_body_size inside the /etc/nginx/nginx.conf file, like so:
client_max_body_size 70M;
After you change all of those values, it might be a good idea to restart the server. If you are using NGINX and PHP-FPM, that might look something like this:
sudo systemctl restart nginx
sudo service php8.3-fpm restart
Why did we increase post_max_size and client_max_body_size to 70 MB if we want to accept files up to 50 MB? It is because when you, for example, convert a 50 MB JPG image to Base64, that Base64 version of the image will be larger than the original binary version of that same image. Approximately, the final size of Base64-encoded binary data is equal to 1.37 times the original data size. If we multiply 50 MB with 1.37, we get 68,5 MB. So, 70 MB should be fine if we are just sending that single image as a payload. If we would like to send some additional data, we could increase those limits even further.
Dealing with memory limit and diving deeper into the framework
So far, so good. In the previous section, we managed to increase the PHP and web server limits in order to send large Base64 files. But if we were to send a POST request to our app, we might encounter these kinds of errors:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 65011744 bytes) in C:\herd-projects\laravel11\vendor\symfony\http-foundation\Request.php on line 1422
Fatal error: Allowed memory size of 146800640 bytes exhausted (tried to allocate 69678520 bytes) in C:\herd-projects\laravel11\vendor\laravel\framework\src\Illuminate\Http\Request.php on line 408
What exactly is going on? The first error is occurring in Symphony's Request class on line 1422. Let's open up that file in our editor. The code on line 1422 is part of the getContent method, and that method looks something like this:
// Symfony\Component\HttpFoundation\Request.php
/**
* Returns the request body content.
*
* @param bool $asResource If true, a resource will be returned
*
* @return string|resource
*
* @psalm-return ($asResource is true ? resource : string)
*/
public function getContent(bool $asResource = false)
{
$currentContentIsResource = \is_resource($this->content);
if (true === $asResource) {
if ($currentContentIsResource) {
rewind($this->content);
return $this->content;
}
// Content passed in parameter (test)
if (\is_string($this->content)) {
$resource = fopen('php://temp', 'r+');
fwrite($resource, $this->content);
rewind($resource);
return $resource;
}
$this->content = false;
return fopen('php://input', 'r');
}
if ($currentContentIsResource) {
rewind($this->content);
return stream_get_contents($this->content);
}
if (null === $this->content || false === $this->content) {
$this->content = file_get_contents('php://input'); // line 1422
}
return $this->content;
}
We can see in DocBlock that the purpose of the function is to return the body content of the request. And on the line 1422, we have this part:
$this->content = file_get_contents('php://input');
where the method is trying to read the contents of our payload. Further, if we check that other error, which is pointing to the Illuminate\Http\Request.php class, we will see that the line 408 is part of the json method, which is in charge of getting the JSON payload of the request:
// Illuminate\Http\Request.php
/**
* Get the JSON payload for the request.
*
* @param string|null $key
* @param mixed $default
* @return \Symfony\Component\HttpFoundation\InputBag|mixed
*/
public function json($key = null, $default = null)
{
if (! isset($this->json)) {
$this->json = new InputBag((array) json_decode($this->getContent(), true));
}
if (is_null($key)) {
return $this->json;
}
return data_get($this->json->all(), $key, $default);
}
On line 408 inside the json function, there is this piece of code:
$this->json = new InputBag((array) json_decode($this->getContent(), true));
in which the JSON payload is being decoded. And there we are able to spot the method we mentioned previously: getContent. So, what we discovered here is that reading and decoding a large Base64 file can require quite a bit of memory. The default memory limit in PHP is usually set to 128 MB. It seems that in order to accept images up to 50 MB, we would need to increase that limit.
Now, we could do that globally by setting the memory_limit value in php.ini:
memory_limit = 256M
But what if we are only accepting these large Base64 images on this single route? Wouldn't it be better if we could target that route only and increase the limit only when those large images might be sent? In order to do that, we need to figure out where the json method we discovered earlier is being called. How do we do that? We could, of course, try to search for the ->json( string inside the Illuminate\Http\Request.php file and see if there are any places where the method is being called directly. If we searched for the string I mentioned, we'd find 3 occurrences of the method being called in that file. We could just add dd('test1') and so on, on all 3 places, and see where the code execution will be stopped. Then we could try to search where the method containing the call to the json method is being called, and so on. That is valid strategy, I guess, but it seems a bit backwards.
What we could also do is start from the beginning. The main file that bootstraps the Laravel framework and is an initial entry point for handling HTTP requests sent to your app is located in the public folder. That file is index.php. If we open the public/index.php file at the bottom, you will notice these lines:
// public/index.php
// Bootstrap Laravel and handle the request...
(require_once __DIR__.'/../bootstrap/app.php')
->handleRequest(Request::capture());
As far as we can see, the requests are being handled in the capture method of the Illuminate\Http\Request class. If we then open up the capture method, we will observe that it is returning a result of the createFromBase static method, which is creating an Illuminate request from a Symfony instance:
// Illuminate\Http\Request.php
/**
* Create a new Illuminate HTTP request from server variables.
*
* @return static
*/
public static function capture()
{
static::enableHttpMethodParameterOverride();
return static::createFromBase(SymfonyRequest::createFromGlobals());
}
Finally, if we continue going down the rabbit hole, we can notice that the createFromBase method is actually calling our json method:
// Illuminate\Http\Request.php
/**
* Create an Illuminate request from a Symfony instance.
*
* @param \Symfony\Component\HttpFoundation\Request $request
* @return static
*/
public static function createFromBase(SymfonyRequest $request)
{
$newRequest = new static(
$request->query->all(), $request->request->all(), $request->attributes->all(),
$request->cookies->all(), (new static)->filterFiles($request->files->all()) ?? [], $request->server->all()
);
$newRequest->headers->replace($request->headers->all());
$newRequest->content = $request->content;
if ($newRequest->isJson()) {
$newRequest->request = $newRequest->json();
}
return $newRequest;
}
Specifically, this part here is important to us, as it says if the request is a JSON request, call the json method and decode the data for us:
if ($newRequest->isJson()) {
$newRequest->request = $newRequest->json();
}
Great, we found it and learned a bit about the request lifecycle. Now, how can we alter this createFromBase method to fit our needs?
Creating the custom request
What we could do is extend the Illuminate\Http\Request class with our own custom class and override the createFromBase method from the parent class with our own altered version. You could place this custom request class anywhere you like; for simplicity and clarity, I'm going to place it in the Overrides folder inside the app folder. So the namespace would be something like this: App\Overrides\Request. Here is how that might look in your project:
And here is the whole code of the class:
// App\Overrides\Request.php
namespace App\Overrides;
use Symfony\Component\HttpFoundation\Request as SymfonyRequest;
use Illuminate\Http\Request as LaravelRequest;
class Request extends LaravelRequest
{
/**
* Create an Illuminate request from a Symfony instance.
*
* @param \Symfony\Component\HttpFoundation\Request $request
* @return static
*/
public static function createFromBase(SymfonyRequest $request)
{
$newRequest = new static(
$request->query->all(), $request->request->all(), $request->attributes->all(),
$request->cookies->all(), (new static)->filterFiles($request->files->all()) ?? [], $request->server->all()
);
$newRequest->headers->replace($request->headers->all());
$newRequest->content = $request->content;
if ($newRequest->isJson()) {
// increase memory limit for large JSON payloads
if($newRequest->getMethod() == "POST" && $newRequest->path() == "api/json") {
ini_set('memory_limit', '256');
}
$newRequest->request = $newRequest->json();
}
return $newRequest;
}
}
The most important change we made is, of course, related to how JSON requests are being handled:
if ($newRequest->isJson()) {
// increase memory limit for large JSON requests
if($newRequest->getMethod() == "POST" && $newRequest->path() == "api/json") {
ini_set('memory_limit', '300M');
}
$newRequest->request = $newRequest->json();
}
Here we are increasing the memory limit only if the request method is POST; it is targeting our /api/json route that is in charge of handling those large images. Great!
Using the custom request
Now, if we were to try to send that POST request with a large Base64 image as a payload, we would notice that our custom request class and the method inside of it aren't being called at all. That is because we need to replace Laravel's Illuminate\Http\Request class with our custom implementation and use the App\Overrides\Request class instead. We need to perform that replacement in the index.php file. There are only two things we need to do, import our custom request class at the top of the file, like so:
use App\Overrides\Request as CustomRequest;
and then just use our custom request class instead of Laravel's request.
// Bootstrap Laravel and handle the request...
(require_once __DIR__.'/../bootstrap/app.php')
->handleRequest(CustomRequest::capture());
Here is how the whole file might end up looking:
use Illuminate\Http\Request;
use App\Overrides\Request as CustomRequest;
define('LARAVEL_START', microtime(true));
// Determine if the application is in maintenance mode...
if (file_exists($maintenance = __DIR__.'/../storage/framework/maintenance.php')) {
require $maintenance;
}
// Register the Composer autoloader...
require __DIR__.'/../vendor/autoload.php';
// Bootstrap Laravel and handle the request...
(require_once __DIR__.'/../bootstrap/app.php')
->handleRequest(CustomRequest::capture());
If needed, we can easily just swap back the original request class. And that's really all there is to it.
Conclusion
We came to the end of the road. In this article, we saw what issues we might encounter when trying to accept large Base64 payloads and how to resolve them. We learned a bit about the request lifecycle and how we can debug and alter some of the framework's inbuilt classes and methods. Since the framework is constantly evolving and being updated, some of the code lines mentioned might change in the future, but the general idea should remain the same. I hope that the things we went through in this article today might help you in the future when you encounter similar issues.
Thank you for taking the time to go through this article.
My name is Goran, I'm a software developer that enjoys working with Laravel and exploring its ecosystem.
For more details, check out the about page.
Find me on LinkedIn, Twitter (X) or send me an email.