When looking to improve the performance of an SSIS package, one of the common recommendations listed in many places is to modify your buffers so that you can load as much data into a single buffer size as possible. According to Microsoft, for optimal performance, SSIS should be configured for maximum memory utilization by having buffers with as many rows as possible without exceeding the internal 100 MB limit. Ok now that you know what Microsoft says about the topic, how does one put it in practice? I thought it might be nice to post the How-To as it is a little complicated.
Buffer Tuning Parameters
There are a couple of default settings that are used for loading data into memory and they all need to line up correctly to keep data from being sent from memory to disk. In order to set the size appropriately, it is best to apply the Price Is Right Axiom.
DefaultMaxBufferSize – the default is 10 MB. The maximum size is 100MB, which SSIS stores as MaxBufferSize. This value can never be any bigger than 100MB, so size it appropriately for it’s environment.
DefaultMaxBufferRows – The default is 10,000 rows. This setting needs to be set the same way that you win with the Price is Right . To win, the default max buffer rows * (times) the row size needs to be as close to DefaultMaxBufferSize as possible without going over.
Size Matters
To figure out what to put in the DefaultMaxBufferRows, one needs to know how big the rows are. Remember to win the value must be as close to the size of DefaultMaxBufferSize without going over. If the buffer is sized too large, the package will be slower as rows will be cached to disk, so make sure you don’t miss by even a little. How big is the row size? There are a number of ways of figuring this out. You can look at each field in the query SSIS is loading, and based on the data type, add up the value of all the fields. Fortunately that isn’t the only way to figure out the row size. If your data source is SQL Server, you are in luck, as the system tables can help to determine what the size is. Here is a sample the query, assuming your table name is Address, which you can run on the AdventureWorks database.
Exec sp_SpaceUsed 'Person.Address'
The Results are
name rows reserved data index_size unused
Address 19614 5960 KB 2784 KB 2688 KB 488 KB
To figure out what the size of your buffer should be for this entire table is to take the number of (data *1024)/ Rows as 100MB is the max size you can set. To calculate the row size, use the formula values 2784 / 19614 * 1024 = 145.346, or 146 bytes per row. If you set DefaultMaxBufferRows to 100MB, which is the maximum and what I recommend in most cases, it is 104857600 bytes is the Buffer Size. Buffer Size/ Row size = DefaultMaxBufferRows. 104857600 / 146 = 718202.73 so set the DefaultMaxBufferRows to 728203 If you are using the columns, you can get the same information by looking at the syscolumns. By using the column length, it is relatively easy to figure out what the appropriate size of your buffer should be, by adding up the column lengths. One word of caution. I do not wish to imply that because the information is available on a per table basis one should pick Table or View in the SSIS source. Au contraire. Always access the data by using a Select statement as it performs better.
Validate
To ensure that you have improved the buffer size performance, check it. After you are done with the settings, Enable logging on the data flow task, and select the BufferSizeTuning event to see how many rows are contained in each buffer.
Please feel free to drop me a line if you find this helpful.
Yours Always
Ginger Grant
Data aficionado et SQL Raconteur
Pingback: SQL 2016 SSIS new feature AutoAdjustBufferSize – Another SQL Geek
Pingback: Size does matter: SSIS DefaultBuffer | Michellea David, Sr. DBA
Thanks Ginger,
This was very Helpful for me to begin tuning some of our many SSIS Packages.
Loyd C.
Great stuff!
Wanted to note the increase in maxaximum buffer size, from 100mb through SQL Server 2014 to 2^31-1 bytes (2GB – 1 byte) in SQL Server 2016.
https://docs.microsoft.com/en-us/sql/integration-services/data-flow/data-flow-performance-features
Denzil Ribeiro discusses the columnstore motivation for the SSIS 2016 max buffer size increase here.
https://blogs.msdn.microsoft.com/sqlcat/2016/02/29/sql-server-2016-ssis-data-flow-buffer-auto-sizing-capability-benefits-data-loading-on-clustered-columnstore-tables/
Lonny —
Great point! Thanks for bringing it up.
Regards,
Ginger
Hi Ginger,
I have been a regular reader of your blog. I love your writing. You are so simple & genuine and that’s what makes me connect with your writing. Thanks for sharing.
Thank you very much for the lovely complement.
Regards,
Ginger
This is Gem of an information..thank you so much for sharing.
Very helpful information. Clearly written. Thank you so much for the sharing!
Hi Ginger – first off, this has been useful as I’m trying to get my head around this for our 2008 R2 SSIS environment, so thank you 🙂
I do have one question; you said “104857600 / 146 = 718202.73 so set the DefaultMaxBufferRows to 728203”, which seems counter-intuitive to me. If 718202.73 is the max, why are you adding 10,000 and rounding up to 728203?
I didn’t see anyone ask about this in the comments so I’m probably missing something, but I kind of expected that with 718202.73 the DefaultMaxBufferRows would be set to 718202? I.e. close but not over?
Cheers,
Dave
Dave —
Thanks for your comment. In answer to your question, you are correct. It is close but not over.
Regards,
Ginger