To get Apache to behave when it is running as the back end of any sort of load balancing mechanism (be it nginx, squid, or anything else) you will need a module that knows how to handle X-Forwarded-For and other headers correctly. Grab it from https://github.com/gnif/mod_rpaf or from your trusty package manager.
apt-get install libapache2-mod-rpaf
The default configuration in Debian is acceptable if your load balancer is running locally (such as for caching or routing purposes), otherwise you will want to edit the "RPAFproxy_ips" setting to contain the hosts that are doing the load balancing. Also ensure that you set "RPAFheader" to "X-Forwarded-For" to get the desired effect. Once you are done, restart Apache and check its logs to see if it is recognizing the IPs of incoming connections correctly. Instead of all hits being from localhost, they should now show up as being from the actual client IP.
The best part of mod_rpaf is that you do not need to configure your software stack to handle the X-Forwarded-For header. It is by far the fastest way to add this support to your software stack.