Wednesday, March 12, 2014

Custom Excel Reports in SSIS with C#

<edit 2014-05-22>
you must add the folder
C:\Windows\SysWOW64\config\systemprofile\Desktop (C:\Windows\System32\config\systemprofile\Desktop if it is 32bit)
or running this job will fail under SQL Server Agent login
scheduled as a SQL Server Agent job.
</edit>


If you need more customized excel reports (What exec or salesperson these days are happy with PDF reports anymore), especially custom templates with multiple worksheets (different report per worksheet), you can do this with a script component in SSIS fairly easily.

I'm not going to cover making an SSIS project, data flow task, or ADO NET (SQL statement) source as those are pretty easy and there are plenty of guides on the net already.

This is dealing specifically with the "script component" as Destination from the SSIS Toolbox as:










You will need to select the ADO NET columns you need for the new script component as usual:

For this simple example I am only selecting 4 columns from the database.
As an addition step (not required), I added a "filename" user variable to the project:



What good is an excel report if you can't email it to someone?
In this example we are creating a new xlsx file from scratch (not from an existing formatted template) and naming it with the date and time in the filename for historical purposes (archiving distributed reports is one of our business requirements)

Add the "filename" variable to the script component on the first section under readwrite variables:




I use C# as the scripting language because I prefer C# to VB, but you could translate the script to VB if you like.

Now click on Project->Add Reference:

You will need to add two references:
COM section
Microsoft Excel 11.0 Object Library (or version available)
Assemblies section
Microsoft.CSharp

Then you can copy and past the script below and edit it.
NOTE: You will HAVE to edit the script. Parts of it are generic sure, but you must set the column
names in the variable declaration, the starting line number, the number of columns, etc. to exactly match
what input columns you are passing into the script component.
Also, I am saving the excel files to my C:\SSIS folder in my VM. You will need to change that to suit the machine you are running it from.
I'm only using one worksheet. If you are outputting multiple worksheets you will have to add the logic for that yourself.

If you have any questions drop me a line.


------------------------------------------------------------

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using Microsoft.Office.Interop.Excel;

#endregion

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    Microsoft.Office.Interop.Excel.Application    ivar_Excel                = new Microsoft.Office.Interop.Excel.Application();
    Microsoft.Office.Interop.Excel.Workbook        ivar_Workbook;
    Microsoft.Office.Interop.Excel.Worksheet        ivar_Worksheet;
    Microsoft.Office.Interop.Excel.Range            ivar_Range;

    //                    il_Row should be far enough down to skip your Header section
    long                il_StartDataRow        =                10;
    long                il_Row                    =                10;
    long                il_Rowcount                =                0;

    // YOU MUST FILL THIS PART IN!
    String[]            iS_ColumnNames            =                {"Employee ID","Name","Title","Employee Number"};
    String[]            iS_WorksheetNames        =                {"Employee Data"};


    public override void PreExecute()
    {
            base.PreExecute();
       
            // CREATE a new excel file to output to and saveas
            ivar_Excel.Visible                =    false;
          
            ivar_Workbook                        =    ivar_Excel.Workbooks.Add();
            ivar_Worksheet                        =    ivar_Workbook.Sheets[1];

            // set the name of the worksheet
            ivar_Worksheet.Name                =        iS_WorksheetNames[0];
            ivar_Worksheet.Cells[1,1]        =        "Date:";
            ivar_Worksheet.Cells[1,2]        =        DateTime.Now.ToString("u").Replace( "Z","" );

            // set the 1st row Header Text
            for(    long  ll_Index=1;
                    ll_Index <= iS_ColumnNames.Length;
                    ll_Index++ )
            {
                    ivar_Worksheet.Cells[il_StartDataRow-1,ll_Index]        =        iS_ColumnNames[ ll_Index -1];
            };

            //change column widths
            for(    long ll_Index=1;
                    ll_Index <= iS_ColumnNames.Length;
                    ll_Index++ )
            {
                    ivar_Range    =    ivar_Worksheet.Range[    ivar_Worksheet.Cells[ il_StartDataRow-1,ll_Index ],
                                                                        ivar_Worksheet.Cells[ il_StartDataRow-1,ll_Index ] ];
                    ivar_Range.EntireColumn.ColumnWidth =    50;
            };

            // title row color and bold
            ivar_Range    =    ivar_Worksheet.Range[    ivar_Worksheet.Cells[ il_StartDataRow-1, 1],
                                                                ivar_Worksheet.Cells[ il_StartDataRow-1, iS_ColumnNames.Length]  ];
            ivar_Range.Font.Color                                    =    XlRgbColor.rgbBlack;
            ivar_Range.Interior.Color                                =    XlRgbColor.rgbYellow;
            ivar_Range.Font.Bold                                        =    true;
            ivar_Range.EntireRow.RowHeight                        =    20;


    }

    public override void PostExecute()
    {
            string    ls_filename;


            base.PostExecute();

            //change column colors
            for(    long  ll_Index=1;
                    ll_Index <= iS_ColumnNames.Length;
                    ll_Index++ )
            {
                    ivar_Range    =    ivar_Worksheet.Range[    ivar_Worksheet.Cells[ il_StartDataRow,ll_Index ],
                                                                        ivar_Worksheet.Cells[ il_Row,ll_Index ] ];
                    ivar_Range.ColumnWidth =    35;
                    ivar_Range.Font.Color                    =    XlRgbColor.rgbBlack;
                    ivar_Range.Interior.Color                =    XlRgbColor.rgbWhite;
            };
          

            // set font bold and yellow for summary row
            ivar_Range                                                =    ivar_Worksheet.Range[    ivar_Worksheet.Cells[ il_Row, 1 ],
                                                                                                            ivar_Worksheet.Cells[ il_Row, iS_ColumnNames.Length ] ];
            ivar_Range.Font.Color                                =    XlRgbColor.rgbBlack;
            ivar_Range.Interior.Color                            =    XlRgbColor.rgbYellow;
            ivar_Range.Font.Bold                                    =    true;
            ivar_Range.EntireRow.RowHeight                    =    20;

            // rowcount
            ivar_Worksheet.Cells[ il_Row, 1 ]                =    "Rows: ";
            ivar_Worksheet.Cells[ il_Row, 2 ].Formula        =    "=COUNTA(B2:B" + il_Rowcount.ToString() + ")";

            // build save filename with date in it      
            ls_filename    =        DateTime.Now.ToString("u") + ".xlsx";
            ls_filename    =        ls_filename.Replace( ":","-" );
            ls_filename    =        ls_filename.Replace( "Z","" );

            ls_filename =        "\\\\brucemdev\\C$\\SSIS\\" + ls_filename;

            //System.Windows.Forms.MessageBox.Show(ls_filename);
            ivar_Workbook.SaveAs( ls_filename );
            ivar_Workbook.Close();
            ivar_Excel.Visible = false;

            Variables.filename        =        ls_filename;



    }

    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
            long        ll_columncount;

            // THIS SECTION IS NOT DYNAMIC YOU MUST CODE IT FOR EVERY COLUMN
            ivar_Worksheet.Cells[il_Row,1]        =   Row.employeeid.ToString();
            ivar_Worksheet.Cells[il_Row,2]        =   Row.name.ToString();
            ivar_Worksheet.Cells[il_Row,3]        =   Row.title.ToString();
            ivar_Worksheet.Cells[il_Row,4]        =   Row.number.ToString();
          

          


            il_Row++;
            il_Rowcount++;


    }

}


You will end up with an excel output that looks like:

Pretty crude as far as a report you would send to an end user, I know, but you can add additional formatting in the script as desired. ( title, subtitle, description, arguments, etc)




Tuesday, July 30, 2013

zfs on ubuntu locks up if you overcommit arc

Hey just a quick note.. be careful what you ask for.
(this applies to the zfsonlinux kernel mode zfs, not the fuse variety)
From a couple different servers, it seems you need to leave (at a rough guess) 4GB of memory free for ubuntu or your zpool scrub can hang your system if you manually set your zfs_arc_max parameter too high.

Safe config (has been working for me anyway) for a 10GB VM:

root@ubuntuzfs03:~# ~/zfs_show.sh
config
---------------------
options zfs zfs_arc_max=6000000000 zfs_arc_meta_limit=4900000000 zfs_arc_min=5900000000



runtime values
---------------------
c_min                           4    5900000000
c_max                           4    6000000000
size                            4    43729016
hdr_size                        4    1011296
data_size                       4    41999872
other_size                      4    717848
anon_size                       4    16384
mru_size                        4    14393344
mru_ghost_size                  4    0
mfu_size                        4    27590144
mfu_ghost_size                  4    16384
l2_size                         4    0
l2_hdr_size                     4    0
duplicate_buffers_size          4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    36782200
arc_meta_limit                  4    4900000000
arc_meta_max                    4    36782200
root@ubuntuzfs03:~#

I was running with 8GB of ARC and my zpool scrub was crashing. Anyway, just wanted to share, as its not obvious why the zpool scrub was locking up the system, but it seems to be something to do with the kernel not being able to allocate memory.


Wednesday, July 3, 2013

Breaking up with your tape drive



Dear tape... it’s not you it’s me. I want to see other storage

If you’re like me, backing up to tape for small to medium sized businesses (SMB’s) just doesn’t make sense anymore. The high cost of a tape drive, even higher for a tape library, and the high cost per MB for each tape make backups an expensive (but necessary) job with $0 ROI.
                If you had a satellite office with a reasonably fast VPN connection between them, you could easily consider replicating your data (one way). This would have the advantage of having offsite backups and disaster recovery, but it is assuming that you have the money for 2 of every piece of hardware it takes to run your production systems. And two datacenters operating 24/7 is an additional expense.
                You can pay for cloud storage and replicate your VM’s offsite to Amazon or one of those file hosting services, but at the price you pay per MB that’s more expensive than tape, and doesn’t work well for large volumes of data anyway (SQL Server backups, mail backups, fileserver backups, etc).
                So, if you work in a SMB with one office and a limited budget, but you still want to be rid of the hassle of tapes, consider a removable disk storage alternative paired with a non removable drive. More specifically: any machine (desktop/server) with a PCI-e 1X slot and empty SATA drive bay will work, but gigabit NIC as close to the production server(s) you are backing up is a definite requirement. If you are backing up large quantities of data, you might go for option 2.

Backup Server O/S options:
Option 1: Ubuntu 12.04 with 1gb NIC
·         free
·         rsync
·         ZFS support for mirroring (www.zfsonlinux.org), send/receive replication, snapshots
·         supported by Veeam 6.5 B&R as a backup repository

Option 2: Hyper-V server 2012
·         free
·         NIC teaming in either switch independent or LACP modes with different brand NICs or even a mixture of plug-in NIC cards and MB NIC ports
·         can run Ubuntu 12.04 as a VM (option 1) to handle backups, plus other VM’s to get more use out of server grade hardware
·         also use as for VM replicas/disaster recovery

Option 3: Windows Server 2012
·         not free. Standard edition (only 1 VM included) will run you $900 or so.
·         NIC teaming in either switch independent or LACP modes with different brand NICs or even a mixture of plug-in NIC cards and MB NIC ports
·         Can run in parallel with existing tape backup jobs or as a supplement to tape backup jobs (if for some reason you are not able to replace all of them)

If you are at a SMB you probably would chose option 1 or option 2 as they are the most cost effective.
The strategy here is to run a backup job (say through Veeam B&R, or Backup Exec, or whatever backup software you are using) to the internal hard drive on the backup server. Then (at a later time) rsync or robocopy any changed blocks/files to the USB external storage. The external storage can be removed for storage in a safe, offsite/whatever. You can even swap out external storage like you do tapes. For our experiment, we went with a 4TB USB 3.0 drive paired with a PCI-e 1X controller card. Both are recognized with no problems in Ubuntu and seem to deliver reasonable speeds. We do a “monthend” backup where we take the 4TB drive out of the safe, do a reverse incremental backup with Veeam, and put the tape back in the safe. The total cost of this setup (considering we re-used a desktop PC as the Ubuntu server option 1) at the time of writing this blog:  

$159.99 for the 4TB seagate drive (as listed on newegg)
$26.99 for the StarTech 2 Port PCI Express SuperSpeed USB 3.0 Card Adapter Model PEXUSB3S2
$186.98


Even if you wanted to buy multiple 4TB drives and rotate them out on a weekly or daily basis to a safe or offsite location its still cheaper than buying a server, windows server license, backup exec (or other) license, and a tape drive or tape library. I will update this blog at a later date if we run into any issues with this setup.

Thursday, March 14, 2013

Using VM Replication to create a test environment

Often in IT you find yourself needing to test something. The requirements for that can range from the simple example of needing to install or upgrade software on one server to the seriously complex side of full regression tests using a software test suite that encompasses client machines, multiple servers and databases.

Before the advent of virtualization, this usually meant the system administrator had to support multiple servers for the same function. In other words, you might have 3 or 4 servers for a single database because of having to support development, qa, and user acceptance testing requirements.

Now, with hyper-v (or ESX), Veam Backup and Replication, and ZFS those tasks have been made significantly easier. Additionally, you can accomplish all of those things with less hardware and operating expenses.

We use Veam as our replication and backup software for our production Hyper-V virtual machines (VM's). The nice part of that setup (aside from the fact that its a solid and reliable product) is the way Veeam licenses their software. You pay per socket on the server you are backing up from. You do not pay for replication or backup targets. So, in other words, if you license your production server, you can back it up to as many destinations as you care to. That works out especially well for making a testing/qa/acceptance testing copy of every single one of your VM's to a (usually less expensive) test hypervisor. The only significant requirement for the test hypervisor is adequate memory. You don't even necessarily need a RAID array, a number of VM's can (slowly) run off the same single SATA drive.

Creating the replication job in Veeam is pretty straightforward. You select the VM's you want to replicate, the machine you want to replicate them to, and the default suffix to add (it puts _replica by default). I would recommend changing that to _development or _qatesting or something more indicative of what you're going to use it for.

Not a requirement, but it would be very beneficial (as you will see in a minute) if you could also make the target replication directory an iSCSI target on a ZFS datastore. The reason for this, is that managing snapshots on one test VM can easily be managed in Hyper-V manager. But trying to synchronize the snapshots and performing multiple rollbacks and restarts on 25 VM's would be a pain to say the least. If you had the option of snapshoting and rolling back the entire iSCSI target (easily done with a ZFS SAN backend) you can do multiple regression tests in a quick and painless (at least less painful) way.

The only “gotcha” or issue to work around here, is that you have to use a private virtual switch in the test hypervisor. This will keep you from having to reassign IP addresses, computer names, leave the domain, re-join a test domain, etc. If you live in an AD (active directory) environment, you really need a VM copy of your domain controller that is part of the replicated VM's. Not having to change a single setting on any of the servers or client machines is really really nice. Because you are using a private virtual switch, you have to connect to a client test VM or test server running your application or test software through hyper-v manager. Incovenient yes, but to me a small price to pay to get that much bang for your buck.

If you need further explanations or step by steps with screenshots of any part of that, leave me a comment and I can expand this blog post. No, I don't work for Veeam I just happen to love their B&R product :)

Tuesday, January 15, 2013

compression, dedup, and compression + dedup test results




So, I ran some test results with some VHD and VHDX vm files I had from a backup, and
 it was interesting to see the results of deduplication vs compression vs both at the same time.

I did 3 tests, each time copying the same set of 166 GB worth of VHD and VHDX backup files.

First option was dedup only, RECSIZE=16K
This required at least 2.6 GB of RAM in your arc_meta_limit and had a poor dedup ratio.

Second option was compression only, COMPRESSION=LZJB
This does use arc_meta_limit, obviously, but its not imperative that you be able to fit all of it in memory at once.

Third option was dedup on AND compression on. You can see that the compression interfered with the deduplication ratio. I would assume that is partly because the parts of the VHD that are highly compressible are also the ones that are dedup-able. The interesting thing here is that turning compression AND dedup on resulted in a faster write speed than just dedup. I would assume because it is trying to dedup 77.4G of data instead of deduping 166G of data.  The deletion time was also faster.

You can see the detailed results from the XLS screenshot:



The conclusion here (imho) is that dedup is VERY situational and typically is not going to be worth your while compared to LZJB or GZIP-X compression.

I supposed if you are storing multiple copies of the exact same files dedup + compression would come in handy, but I can't think of any situations that would come into play where a snapshot + clone wouldn't work better.

If you have a specific situation where dedup or dedup + compression wins over just compression for you, please let me know what that was.