Setting quota?

I think I have found the solution
I am using ASP.NET version so it may differ for other languages.

Followings are what you must edit:

Settings.ConfigFile.cs

Add public int QuotaSize; //this is in MegaByte
Add public string QuotaSetFolder;
Also add in constructor as well, and do something like QuotaSetFolder = BaseUrl;

Add following codes at the bottom so you can access later

        public int QuotaSize
        {
            get { return Settings.ConfigFile.Current.QuotaSize; }
        }

        public string QuotaSetFolder
        {
            get { return Settings.ConfigFile.Current.QuotaSetFolder; }
        }

Add a new error code (code number can be anything)
public const int ExceedsQuota = 550;

CKFinder.Connector.CommandHandlers.FileUploadCommandHandler.cs
Add a code to check filesize of uploaded file, and directory size then compare (I devide the ContentLength by 1024 twice to convert bytes --> kilo bytes --> mega bytes).

        if ((oFile.ContentLength / 1024 / 1024) + GetDirSize(new System.IO.DirectoryInfo(Config.Current.QuotaSetFolder)) > Config.Current.QuotaSize)
                            ConnectorException.Throw( Errors.ExceedsQuota );

and also add a method to calculate directory size

        public long GetDirSize(System.IO.DirectoryInfo dirinfo)
        {
            long totalSize = 0;

            foreach (System.IO.FileInfo fileinfo in dirinfo.GetFiles())
                totalSize += fileinfo.Length / 1024 / 1024; //in MB

            foreach (System.IO.DirectoryInfo dirinfosub in dirinfo.GetDirectories())
                totalSize += GetDirSize(dirinfosub);

            return totalSize;
        }

You also need to modify the language file (and add error message).
For en.js, around line 107, there is a section to declare error messages. At the end of it, add error 550 like following

500 : 'The file browser is disabled for security reasons. Please contact your system administrator and check the CKFinder configuration file.',
501 : 'The thumbnails support is disabled.',
550 : 'Exceeds Quota'
},

That's it
Now in your config.ascx, specify QuotaSize and QuotaSetFolder and you are all set.
Example

   public override void SetConfig()
   {
      QuotaSize = 1;
      QuotaSetFolder = @"C:\Inetpub\wwwroot";