[Tutor] airflow dag

[Alan Gauld]

On 25/05/17 13:15, shubham goyal wrote:
> He guys,
> 
> I want to ask that can we pass the parameters as commandline arguments in
> airflow when we are triggering the dag and access them inside the dag's
> python script/file.

I've no idea what a dag is.
This list is for people learning Python as a programming
language but it has no knowledge(necessarily) of other
environments that use Python. So you need to tell us more.

In generic terms Python can read command line arguments
in the sys.argv list. So you could try adding a line like

import sys
...
print sys.argv


And see if your arguments appear there.
Otherwise you'll need to find a dsag forum and try asking there.
Or get very lucky and find someone on this list that speaks dag.

If they are present in sys.aergv then there are several modules
that can parse the arguments looking for option flags etc.
argparse is the current official recommendation.


> script:
> 
> from airflow import DAG
> from datetime import datetime,timedelta
> default_args = {
>     'owner': 'airflow',
>     'depends_on_past': False,
>     'start_date': datetime.now(),
>     'email': ['airflow at airflow.com'],
>     'email_on_failure': False,
>     'email_on_retry': False
> }
> MAIN_DAG='check_dag'
> dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
> schedule_interval=None)
> 
> with open(file, "r") as f:
>     payload = f.read()  # Reading the json data from a file
>     SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
>         task_id='cluster_create',
>         method='POST',
>         http_conn_id='qubole_default',
>         # for directing to https://qa.qubole.net/api
>         endpoint='/v2/clusters?auth_token=%s' % (passwd),
>         data=payload,
>         headers={"Content-Type": "application/json"},
>         params={'auth_token': passwd},
>         response_check=lambda response: True if response.status_code == 200
> else False,
>         dag=dag

I'm not sure what you think the line above does but
in normal Python it would have zero effect.

> like this here i am trying to create a cluster but i need to pass password
> as cli arguments when i trigger the dag. can we do that. please help.

You need to explain what you mean bearing in mind we don't
know anything about dag, or clusters.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos