Connect to a DataSet Variable as a Source?

Hello. I’m new to SSIS. This forum and Kirk Haselden’s book are my teachers. I’m having a hard time grasping something basic to get started defining a set of packages to automate the ETL process, however, and perhaps I’m simply misunderstanding the best practices of SSIS.

I have source data in two different transactional databases, and use OleDb connection managers (and OleDb Source components in the Data Flow) to extract the data. I use a Script Task and several Lookup widgets in the Data Flow to transform the data, and output each to two different package-scope DataSet variables.

How do I join these two datasets in a third Data Flow task for loading into my data warehouse? It seems I can iterate through them in the Control Flow, but I can’t write a query against them in the Data Flow, since there is no connection manager that allows me to “connect” to a package-level variable. Should I instead be storing my extracted, transformed data in temporary database tables, and then joining these to do the final load?

Any advice greatly appreciated. Thanks in advance.

MilwaukeeCharlie wrote:

Hello. I’m new to SSIS. This forum and Kirk Haselden’s book are my teachers. I’m having a hard time grasping something basic to get started defining a set of packages to automate the ETL process, however, and perhaps I’m simply misunderstanding the best practices of SSIS.

I have source data in two different transactional databases, and use OleDb connection managers (and OleDb Source components in the Data Flow) to extract the data. I use a Script Task and several Lookup widgets in the Data Flow to transform the data, and output each to two different package-scope DataSet variables.

How do I join these two datasets in a third Data Flow task for loading into my data warehouse? It seems I can iterate through them in the Control Flow, but I can’t write a query against them in the Data Flow, since there is no connection manager that allows me to “connect” to a package-level variable. Should I instead be storing my extracted, transformed data in temporary database tables, and then joining these to do the final load?

Any advice greatly appreciated. Thanks in advance.

Charlie,

You're absolutely correct, there is no source adapter for these and with good reason too.

There is a workaround though:

Recordsets instead of raw files

(http://blogs.conchango.com/jamiethomson/archive/2006/01/04/SSIS_3A00_-Recordsets-instead-of-raw-files.aspx)

Another option (and the one I would use for performance reasons as explained here: http://blogs.conchango.com/jamiethomson/archive/2006/06/28/SSIS_3A00_-Comparing-performance-of-a-raw-file-against-a-recordset-destination.aspx) is to use raw files

By the way, the correct nomenclature is "components", not "widgets"

Good luck!

-Jamie

Conflict Resolution: Is defining local subscription enough: How to get log of lost record.

Hello All,
I am very much thankful to you all for providing much needed help, and
once again I am at the group with my query. In my replication
implementation in which I have around 100 subscriber I need to decide
what conflict resolution policy should be employed.
We, in our implementation just want that publisher always wins in case
if there is any conflict. Is it enough to create a local subscription
to acheieve this.
Which will ensure that in case of any conflict with the publisher
publisher always wins the conflict.
Also we want that losing party should get log of inforamtion about the
lost record, is there any way to acheive this?
Can you define what you mean by a local subscription?
The best way to alert the user of conflicts is to use the com object
C:\Program Files\Common Files\Microsoft Shared\Database
Replication\WZCNF.DLL
This will display a list of the conflicts. You will then have to poll for
them and send out alerts.
Subscriptions managed by Windows Synchronization Manager using the
interactive resolved will allow the user to manage the conflict when WSM
runs.
Hilary Cotter
Looking for a SQL Server replication book?
http://www.nwsu.com/0974973602.html
"Ruchir" <ruchirdhar@.gmail.com> wrote in message
news:88216eb7.0501172243.6e17e067@.posting.google.c om...
> Hello All,
> I am very much thankful to you all for providing much needed help, and
> once again I am at the group with my query. In my replication
> implementation in which I have around 100 subscriber I need to decide
> what conflict resolution policy should be employed.
> We, in our implementation just want that publisher always wins in case
> if there is any conflict. Is it enough to create a local subscription
> to acheieve this.
> Which will ensure that in case of any conflict with the publisher
> publisher always wins the conflict.
> Also we want that losing party should get log of inforamtion about the
> lost record, is there any way to acheive this?

Conflict Resolution: Is defining local subscription enough: How to get log of lost record.

Publisher always wins is the default, so you don't need
to do anything to achieve this.
To get notification of a conflict, you could create a
polling routine (there's no alert that I know of).
HTH,
Paul Ibison SQL Server MVP, www.replicationanswers.com
(recommended sql server 2000 replication book:
http://www.nwsu.com/0974973602p.html)
hi Paul,
Thanks for you reply, I have few more queries
1. Where is this conflict information specially the losing row information is stored, at the publisher side or at the subscriber side? if losing row information is maintained at the publisher side is there any way so that we can store this information at the publisher side.
2. As you talked about creating a polling routine to get notification of conflict, in my implementation I just want a log file which contain information about the losing row nothing else, will the polling routine you are referring to can do the same and alsoI have not much information about how to write polling routine can you throw some light on it?
regards.
Ruchir.

Quote:

Originally posted by Paul Ibison
Publisher always wins is the default, so you don't need
to do anything to achieve this.
To get notification of a conflict, you could create a
polling routine (there's no alert that I know of).
HTH,
Paul Ibison SQL Server MVP, www.replicationanswers.com
(recommended sql server 2000 replication book:
http://www.nwsu.com/0974973602p.html)

|||Have a look at the tables
conflict_<PublicationName>_<ArticleName>_usertable name
and
MSmerge_delete_conflicts
at the publisher.
The polling routine would be not much more than a job
that runs regularly and does a select from this table,
then sends the results in an email as an attachment. The
origin_datasource column could be used to determine the
correct recipient.
HTH,
Paul Ibison SQL Server MVP, www.replicationanswers.com
(recommended sql server 2000 replication book:
http://www.nwsu.com/0974973602p.html)